We Keep Coding

A dictionary that can save its elements accessed less often to a disk

In my application I use a dictionary (supporting adding, removing, updating and lookup) where both keys and values are or can be made serializable (values can possibly be quite large object graphs). I came to a point when the dictionary became so large that holding it completely in memory started to occasionally trigger OutOfMemoryException (sometimes in the dictionary methods, and sometimes in other parts of code).
After an attempt to completely replace the dictionary with a database, performance dropped down to an unacceptable level.
Analysis of the dictionary usage patterns showed that usually a smaller part of values are "hot" (are accessed quite often), and the rest (a larger part) are "cold" (accessed rarely or never). It is difficult to say when a new value is added if it will be hot or cold, moreover, some values may migrate back and forth between hot and cold parts over time.
I think that I need an implementation of a dictionary that is able to flush its cold values to a disk on a low memory event, and then reload some of them on demand and keep them in memory until the next low memory event occurs when their hot/cold status will be re-assessed. Ideally, the implementation should neatly adjust the sizes of its hot and cold parts and the flush interval depending on the memory usage profile in the application to maximize overall performance. Because several instances of a dictionary exist in the application (with different key/value types), I think, they might need to coordinate their workflows.
Could you please suggest how to implement such a dictionary?

Compile for 64 bit, deploy on 64 bit, add memory. Keep it in memory.
Before you grown your own you may alternatively look at WeakReference http://msdn.microsoft.com/en-us/library/ms404247.aspx. It would of course require you to rebuild those objects that were reclaimed but one should hope that those which are reclaimed are not used much. It comes with the caveat that its own guidleines state to avoid using weak references as an automatic solution to memory management problems. Instead, develop an effective caching policy for handling your application's objects.
Of course you can ignore that guideline and effectively work your code to account for it.
You can implement the caching policy and upon expiry save to database, on fetch get and cache. Use a sliding expiry of course since you are concerned with keeping those most used.
Do remember however that most used vs heaviest is a trade off. Losing an object 10 times a day that takes 5 minutes to restore would annoy users much more than losing an object 10000 times which tool just 5ms to restore.
And someone above mentioned the web cache. It does automatic memory management with callbacks as noted, depends if you want to lug that one around in your apps.
And...last but not least, look at a distributed cache. With sharding you can split that big dictionary across a few machines.

Just an idea - never did that and never used System.Runtime.Caching:
Implement a wrapper around MemoryCache which will:
Add items with an eviction callback specified. The callback will place evicted items to the database.
Fetch item from database and put back into MemoryCache if the item is absent in MemoryCache during retrieval.
If you expect a lot of request for items missing both in database and memory, you'll probably need to implement either bloom filter or cache keys for present/missing items also.

I have a similar problem in the past.
The concept you are looking for is a read through cache with a LRU (Least Recently Used or Most Recently Used) queue.
Is it there any LRU implementation of IDictionary?
As you add things to your dictionary keep track of which ones where used least recently, remove them from memory and persist those to disk.

asp.net session variables to store all global data

I have inherited a project from a developer who was rather fond of session variables. He has used them to store all sorts of global stuff - datatables, datasets, locations of files, connection strings etc. I am a little worried that this may not be very scalable and we do have the possibility of a lot more users in the immediate future.
Am I right to be concerned, and if so why?
Is there an easy way to see how much memory this is all using on the live server at the moment?
What would be the best approach for re-factoring this to use a better solution?

Yes, I would say that you do have some cause for concern. Overuse of session can cause a lot of performance issues. Ideally, session should only be used for information that is specific to the user. Obviously there are exceptions to this rule, but keep that in mind when you're refactoring.
As for the refactoring itself, I would look into caching any large objects that are not user-specific, and removing anything that doesn't need to be in session. Don't be afraid to make a few trips to the database to retrieve information when you need it. Go with the option that puts the least overall strain on the server. The trick is keeping it balanced and distributing the weight as evenly as possible across the various layers of the application.

It was probably due to poor design, and yes you should be concerned if you plan on getting heavier traffic or scaling the site.
Connection strings should be stored in web.config. Seems like you would have to do some redesigning of the data-layer and how the pages pass data to each other to steer away from storing datatables and datasets in Session. For example, instead of storing a whole dataset in Session, store, or pass by url, something small (like an ID) that can be used to re-query the database.

Sessions always hurt scalability. However, once sessions are being used, the impact of a little bit more data in a session isn't that bad.
Still, it has to be stored somewhere, has to be retrieved from somewhere, so it's going to have an impact. It's going to really hurt if you have to move to a web-farm to deal with being very successful, since that's harder to do well in a scalable manner. I'd start by taking anything that should be global in the true sense (shared between all sessions) and move it into a truly globally-accessible location.
Then anything that depended upon the previous request, I'd have be sent by that request.
Doing both of those would reduce the amount they were used for immensely (perhaps enough to turn off sessions and get the massive scalability boost that gives).

Depending on the IIS version, using Session to store state can have an impact on scaling. The later versions of IIS are better.
However, the main problem I have run into is that sessions expire and then your data is lost; you may provide your own Session_OnEnd handler where it is possible to regenerate your session.

Overall yes, you should be concerned about this.
Session is a "per user" type of storage that is in memory. Looking at the memory usage of the ASP.NET Worker Process will give you an idea of memory usage, but you might need to use third-party tools if you want to dig in deeper to what is in. In addition session gets really "fun" when you start load balancing etc.
ConnectionStrings and other information that is not "per user" should really not be handled in a "per user" storage location.
As for creating a solution for this though, a lot is going to depend on the data itself, as you might need to find multiple other opportunities/locations to get/store the info.

You are right in feeling concerned about this.
Connection strings should be stored in Web.config and always read from there. The Web.config file is cached, so storing things in there and then on Session is redundant and unnecessary. The same can be said for locations of files: you can probably create key,value pairs in the appSettings section of your web.config to store this information.
As far as storing datasets, datatables, etc; I would only store this information on Session if getting them from the database is really expensive and provided the data is not too big. A lot of people tend to do this kind of thing w/o realizing that their queries are very fast and that database connections are pooled.
If getting the data from the database does take long, the first thing I would try to remedy would be the speed of my queries. Am I missing indexes? What does the execution plan of my queries show? Am I doing table scans, etc., etc.
One scenario where I currently store information on Session (or Cache) is when I do have to call an external web service that takes more than 2 secs on average to retrieve what I need. Once I get this data I don't need to getting again on every page hit, so I cache it.
Obviously an application that stores pretty much everything it can on Session is going to have scalability issues because memory is a limited resource.

if memory is the issue, why not change session mode to sql server so you can store session data in sql server which requires little code changes.
how to store session data in sql server:
http://msdn.microsoft.com/en-us/library/ms178586.aspx
the catch is that the classes stored in sql server must be serializable and you can use json.net to do just that.

is application state = system.web.httpcontent.cache?

Is application state (http://msdn.microsoft.com/en-us/library/ms178594.aspx) the same as using the System.Web.Caching API?
i.e.
System.web.httpcontent.current.cache[somekey] ?

The answer is there in your own link. Read it.
However, storing large blocks of data in application state can fill up server memory, causing the server to page memory to disk. As an alternative to using application state, you can use the ASP.NET cache mechanism for storing large amounts of application data. The ASP.NET cache also stores data in memory and is therefore very fast; however, ASP.NET actively manages the cache and will remove items when memory becomes scarce.

No, they are not the same.
The ASP.Net Cache object is specifically optimized for caching content or objects that are short lived or can live for a defined amount of time. It will be cannibalized by the garbage collector if resources need to be freed up, and it is never guaranteed that something you put in the Cache will be there the next time you look.
System.Application is a global collection of key-values that can be used to store information global to all users in a thread-safe way (as long as you utilize it safely). However, nothing will ever be removed from System.Application unless you explicitly remove it.

You probably mean System.Web.Context, not content, and the Cache is different from HttpApplicationState.
The application state is there for items that remain fairly static for the lifetime of the application (unless explicitly removed). As can be read in the page you have linked from, the recommendation is to use Application:
to store small amounts of often-used data that does not change from one user to another.
Cache is more transient in nature and is supposed to be used for content that has much shorter life (seconds to minutes) and removal of items is managed automatically (dependent on configuration).

It's not the same.
If the data
is stable during the life of the application
must always be available and must not be purged
you'd use HttpApplicationState.
If the data
not necessarily is needed for the life of the application
changes frequently
can be purged if needed (for example low system memory)
can be discarded if seldom used
should be invalidated/refreshed under some conditions (dependency rule: time span, date, file timestamp, ...)
then use Cache.
Other important differences:
Large amounts of data may better be stored in Cache, the server then can purge it if low on memory.
Cache is safe for multithreaded operations. Page.Application needs locking.
See also this article on etutorials.org for more details.
And this question:
ASP.NET Page.Cache versus Page.Application storage for data synchronization?

Actually I would say that the main purpose of Application state is backwards compatibility with classic ASP applications, and I would never use it in new ASP.NET apps.
As other respondents have indicated, the Cache is actively managed so that data will be discarded if memory is scarce.
Application state is essentially equivalent to a static Hashtable, with locking semantics that are inherited from classic ASP.
If you need to store static data, it's almost always better to store it as a strongly-typed static field of a class rather than using Application state. If you need locking, use the standard synchronisation mechanisms of .NET.

Caching architecture for search results in an ASP.NET application

What is a good design for caching the results of an expensive search in an ASP.NET system?
Any ideas would be welcomed ... particularly those that don't require inventing a complex infrastructure of our own.
Here are some general requirements related to the problem:
Each search result can produce include from zero to several hundred result records
Each search is relatively expensive and timeconsuming to execute (5-15 seconds at the database)
Results must be paginated before being displayed at the client to avoid information overload for the user
Users expect to be able to sort, filter, and search within the results returned
Users expect to be able to quickly switch between pages in the search results
Users expect to be able to select multiple items (via checkbox) on any number of pages
Users expect relatively snappy performance once a search has finished
I see some possible options for where and how to implement caching:
1. Cache on the server (in session or App cache), use postbacks or Ajax panels to facilitate efficient pagination, sorting, filtering, and searching.
PROS: Easy to implement, decent support from ASP.NET infrastructure
CONS: Very chatty, memory intensive on server, data may be cached longer than necessary; prohibits load balancing practices
2. Cache at the server (as above) but using serializeable structures that are moved out of memory after some period of time to reduce memory pressure on the server
PROS: Efficient use of server memory; ability to scale out using load balancing;
CONS: Limited support from .NET infrastructure; potentially fragile when data structures change; places additional load on the database; significantly more complicated
3. Cache on the client (using JSON or XML serialization), use client-side Javascript to paginate, sort, filter, and select results.
PROS: User experience can approach "rich client" levels; most browsers can handle JSON/XML natively - decent libraries exist for manipulation (e.g. jQuery)
CONS: Initial request may take a long time to download; significant memory footprint on client machines; will require hand-crafted Javascript at some level to implement
4. Cache on the client using a compressed/encoded representation of the data - call back into server to decode when switching pages, sorting, filtering, and searching.
PROS: Minimized memory impact on server; allows state to live as long as client needs it; slightly improved memory usage on client over JSON/XML
CONS: Large data sets moving back and forth between client/server; slower performance (due to network I/O) as compared with pure client-side caching using JSON/XML; much more complicated to implement - limited support from .NET/browser
5. Some alternative caching scheme I haven't considered...

For #1, have you considered using a state server (even SQL server) or a shared cache mechanism? There are plenty of good ones to choose from, and Velocity is getting very mature - will probably RTM soon. A cache invalidation scheme that is based on whether the user creates a new search, hits any other page besides search pagination, and finally a standard timeout (20 minutes) should be pretty successful at weeding your cache down to a minimal size.
References:
SharedCache (FOSS)
NCache ($995/CPU)
StateServer (~$1200/server)
StateMirror ("Enterprise pricing")
Velocity (Free?)

If you are able to wait until March 2010, .NET 4.0 comes with a new System.Caching.CacheProvider, which promises lots of implementations (disk, memory, SQL Server/Velocity as mentioned).
There's a good slideshow of the technology here. However it is a little bit of "roll your own" or a lot of it infact. But there will probably be a lot of closed and open source providers being written for the Provider model when the framework is released.
For the six points you state, a few questions crops up
What is contained in the search results? Just string data or masses of metadata associated with each result?
How big is the set you're searching?
How much memory would you use storing the entire set in RAM? Or atleast having a cache of the most popular 10 to 100 search terms. Also being smart and caching related searches after the first search might be another idea.
5-15 seconds for a result is a long time to wait for a search so I'm assuming it's something akin to an expedia.com search where multiple sources are being queried and lots of information returned.
From my limited experience, the biggest problem with the client-side only caching approach is Internet Explorer 6 or 7. Server only and HTML is my preference with the entire result set in the cache for paging, expiring it after some sensible time period. But you might've tried this already and seen the server's memory getting eaten.

Raising an idea under the "alternative" caching scheme. This doesn't answer your question with a given cache architecture, but rather goes back to your original requirements of your search application.
Even if/when you implement your own cache, it's effectiveness can be less than optimal -- especially as your search index grows in size. Cache hit rates will decrease as your index grows. At a certain inflection point, your search may actually slow down due to resources dedicated to both searching and caching.
Most search sub-systems implement their own internal caching architecture as a means of efficiency in operation. Solr, an open-source search system built on Lucene, maintains its own internal cache to provide for speedy operation. There are other search systems that would work for you, and they take similar strategies to results caching.
I would recommend you consider a separate search architecture if your search index warrants it, as caching in a free-text keyword search basis is a complex operation to effectively implement.

Since you say any ideas are welcome:
We have been using the enterprise library caching fairly successfully for caching result sets from a LINQ result.
http://msdn.microsoft.com/en-us/library/cc467894.aspx
It supports custom cache expiration, so should support most of your needs (with a little bit of custom code) there. It also has quite a few backing stores including encrypted backing stores if privacy of searches is important.
It's pretty fully featured.
My recommendation is a combination of #1 and #3:
Cache the query results on the server.
Make the results available as both a full page and as a JSON view.
Cache each page retrieved dynamically at the client, but send a REQUEST each time the page changes.
Use ETAGs to do client cache invalidation.

Have a look at SharedCache- it makes 1/2 pretty easy and works fine in a load balanced system. Free, open source, and we've been using it for about a year with no issues.

While pondering your options, consider that no user wants to page through data. We force that on them as an artifact of trying to build applications on top of browsers in HTML, which inherently do not scale well. We have invented all sorts of hackery to fake application state on top of this, but it is essentially a broken model.
So, please consider implementing this as an actual rich client in Silverlight or Flash. You will not beat that user experience, and it is simple to cache data much larger than is practical in a regular web page. Depending on the expected user behavior, your overall bandwidth could be optimized because the round trips to the server will get only a tight data set instead of any ASP.NET overhead.

How many DataTable objects should I use in my C# app?

I'm an experienced programmer in a legacy (yet object oriented) development tool and making the switch to C#/.Net. I'm writing a small single user app using SQL server CE 3.5. I've read the conceptual DataSet and related doc and my code works.
Now I want to make sure that I'm doing it "right", get some feedback from experienced .Net/SQL Server coders, the kind you don't get from reading the doc.
I've noticed that I have code like this in a few places:
var myTableDataTable = new MyDataSet.MyTableDataTable();
myTableTableAdapter.Fill(MyTableDataTable);
... // other code
In a single user app, would you typically just do this once when the app starts, instantiate a DataTable object for each table and then store a ref to it so you ever just use that single object which is already filled with data? This way you would ever only read the data from the db once instead of potentially multiple times. Or is the overhead of this so small that it just doesn't matter (plus could be counterproductive with large tables)?

For CE, it's probably a non issue. If you were pushing this app to thousands of users and they were all hitting a centralized DB, you might want to spend some time on optimization. In a single-user instance DB like CE, unless you've got data that says you need to optimize, I wouldn't spend any time worrying about it. Premature optimization, etc.

The way to decide varys between 2 main few things
1. Is the data going to be accesses constantly
2. Is there a lot of data
If you are constanty using the data in the tables, then load them on first use.
If you only occasionally use the data, fill the table when you need it and then discard it.
For example, if you have 10 gui screens and only use myTableDataTable on 1 of them, read it in only on that screen.

The choice really doesn't depend on C# itself. It comes down to a balance between:
How often do you use the data in your code?
Does the data ever change (and do you care if it does)?
What's the relative (time) cost of getting the data again, compared to everything else your code does?
How much value do you put on performance, versus developer effort/time (for this particular application)?
As a general rule: for production applications, where the data doesn't change often, I would probably create the DataTable once and then hold onto the reference as you mention. I would also consider putting the data in a typed collection/list/dictionary, instead of the generic DataTable class, if nothing else because it's easier to let the compiler catch my typing mistakes.
For a simple utility you run for yourself that "starts, does its thing and ends", it's probably not worth the effort.
You are asking about Windows CE. In that particular care, I would most likely do the query only once and hold onto the results. Mobile OSs have extra constraints in batteries and space that desktop software doesn't have. Basically, a mobile OS makes bullet #4 much more important.
Everytime you add another retrieval call from SQL, you make calls to external libraries more often, which means you are probably running longer, allocating and releasing more memory more often (which adds fragmentation), and possibly causing the database to be re-read from Flash memory. it's most likely a lot better to hold onto the data once you have it, assuming that you can (see bullet #2).

It's easier to figure out the answer to this question when you think about datasets as being a "session" of data. You fill the datasets; you work with them; and then you put the data back or discard it when you're done. So you need to ask questions like this:
How current does the data need to be? Do you always need to have the very very latest, or will the database not change that frequently?
What are you using the data for? If you're just using it for reports, then you can easily fill a dataset, run your report, then throw the dataset away, and next time just make a new one. That'll give you more current data anyway.
Just how much data are we talking about? You've said you're working with a relatively small dataset, so there's not a major memory impact if you load it all in memory and hold it there forever.
Since you say it's a single-user app without a lot of data, I think you're safe loading everything in at the beginning, using it in your datasets, and then updating on close.
The main thing you need to be concerned with in this scenario is: What if the app exits abnormally, due to a crash, power outage, etc.? Will the user lose all his work? But as it happens, datasets are extremely easy to serialize, so you can fairly easily implement a "save every so often" procedure to serialize the dataset contents to disk so the user won't lose a lot of work.