We Keep Coding

Why might unmanaged memory account for over 60% of memory used by console application?

I'm profiling memory use with ANTS Memory Profiler 7.0 and noticed that unmanaged memory use is ~193MB (~62%) for a console application that does little more than populate some DTOs from 10 million or so records.
The help text for unmanaged memory says:
The memory is assigned to the parts of the application that aren't running as pure .NET code. This includes the common language runtime itself, graphics buffers and any unmanaged data accessed through P/Invoke or COM+
Why might this figure be so high?

You will inevitable use unmanaged code when accessing a database. The interface to the engine is always code that's been around for a long time, predating .NET and wrapped by managed classes that provide the interop. True for, say, SQL Server and any provider that piggy-backs onto OleDb or ODBC.
These managed classes will always implement IDisposable so you can release the resources consumed by the native provider early. Forgetting to do so is very common and rarely noticed. Other than seeing the process running "heavy", seemingly consuming a lot of handles and unmanaged memory for no good reason. This will be especially the case when the garbage collector does not run frequently enough, something you can see with Perfmon.exe. So beyond not using Dispose, part of the problem can be that you don't do enough work with these DTO objects yet to get enough GC churn.
Review your code and ensure you use Dispose() and the using statement where required.

Why doesn't CLR handle the cleanup code?

I have just started with the .NET framework. Today, I was taught about the IDisposable interface and the dispose() method. I was taught a few things regarding it:
dispose() should contain the cleanup code corresponding to an object(like closing any resources occupied by any objects - files or database connections,etc.)
I was also told that in case we don't do it in the dispose() method, the same could be done in the destructor, but that doesn't ensure immediate execution, and we are left to the mercy of GC.
And if at all we don't provide any cleanup code at all, the GC will forcefully terminate all connections to resources that our objects were holding. Hence, we should handle the cleanup code ourselves.
But I was curious as to why doesn't CLR handle this on it's own? It takes care of Memory Management, it takes care of Garbage Collection. So, it should very well know which Object holds onto which resource(s) and when that Object dies off. So, it should be capable of de-allocating those resources as well?
I asked a few people about it. The answer I was given was that it is because we need to close it gracefully, where as GC closes it forcefully. Is it actually the reason?

In .NET there's much more than managed code that the GC knows about. There's like a huge volume of unmanaged code involved: all the file handles, database connections, network sockets, ... all this is plain ol' unmanaged Win32 code. You can't even believe that in almost every single BCL function you are calling from your pretty C# application, you will be hitting like tons of unmanaged functions written in C++ (and may God forbid VB6) and buried deep into the internals of the OS itself. All those functions are allocating unmanaged memory, handles, ... The managed world doesn't know what happens there.
For example every single time you open a file (FileStream) you are basically calling (behind the scenes of course) the CreateFile unmanaged Win32 function. This function allocates an unmanaged file handle directly from the file system. .NET and the GC has strictly no way of tracking this unmanaged code and everything it does. That's why those classes implement the IDisposable interface. So that you could always wrap their instances in using statements and ensure that the Dispose method is always called, even in the event of an exception, and this as soon as possible. The Dispose method will take care of calling another unmanaged function to clean the mess it created.
So basically the way you could think about the IDisposable interface is the following:
The day when we have an operating system written in a fully managed language (something like Midori for example from Microsoft Research) we will probably no longer need IDisposable as the GC will be able to completely replace it as it will have knowledge of everything that happens within this system.

The point of IDisposable and Dispose() is that you should clean up unmanaged memory. That's memory .NET didn't allocate, which came from outside sources and thus the GC cannot know about it. So it cannot clean it up for you automatically. Essentially that's precisely the difference between managed and unmanaged memory ;-)
Generally you should implement Dispose() to clean up whatever unmanaged resources your class uses and implement the finalizer to call Dispose() too. The finalizer is just a safeguard, though. It will make sure that those resources get cleaned up eventually, if the caller forgets to dispose of your class properly.

The IDisposable interface is there to provide you a way to clean up un-managed resources. The CLR only manages your managed resources for you.
In other words, the CLR only knows how to clean up the things that it manages. If you open connections to the rest of the system (like opening files, database connections, etc.), those are your responsibility and you need to tell the CLR how you want it to clean those up for you.

It can only take care of memory management for .NET objects. Any code that needs to use unmanaged resources (because it interacts with a C++ library, for example) falls outside the garbage collector's bailiwick. All that code needs to be told when to release its resources the old-fashioned way.

There's no way for the .Net framework (and the GC) to know how to release a un-managed resource. All it can do, is destroy the reference your managed code has to the resource. It is a lot better to actually call .Close() on a connection to your database server (thereby telling it that the connection should go back into the poll of available connection), than just destroying the reference, and letting it timeout on it's own after a set amount of seconds.
So whenever possible, use the IDisposable interface when referencing un-managed resources!

IDisposable is used when you don't want the GC to handle that particular artifact. The most common example are connections, or file handles. You don't want to wait for the GC to run before releasing a file, or to close a connection to the database, since you don't know when that will happen.
Most people associate IDisposable with unmanaged resources, which is mostly accurate, but fail to remember that finalizers are the proper .NET way to handle those. IDisposable provides a way of deterministicly disposing if that is important to your program.

The IDisposable interface is simply a convention to allow you to deterministically dispose of managed and unmanaged resources. It alone doesn't replace garbage collection or do anything involving the garbage collector itself.
It is more apparent with unmanaged resources because unless these are handled (either in a finalizer or with deterministic disposal) they will remain as a memory leak until the process ends. With managed memory, if you don't deterministically dispose of the items they will be undeterministically collected (assuming eventual eligibility for collection) by the GC, because they are managed (this is also the reason why the dispose pattern doesn't include managed items in the finalizer route).
IDisposable itself doesn't do anything, it is just a recognised interface (and is supported in code with the using keyword) that people expect to find when handling items that use consumable resources, unmanaged memory, external items, etc.
The CLR cannot possibly know when an external item is finished with. That is entirely dependent on the flow of your application. If you happen to also not know when to dispose an object, the finalizer syntax is useful. If you implement a finalizer on a custom class, the garbage collection process will run this finalizer just prior to final collection. This is your last chance to tidy up after yourself.

we use Dispose in order to dispose unmanaged resssource as file access or connection database, because GC don't have information about this unmanaged ressource.
you can also use Finalize, but it's not performant because you save your ressource in finalisation structure, and GC pass in the end of dispose cycle by this finalisation structure, and it's not performant

Where can I find more detail on disposal of COM resources in .NET?

The background for my question is this: I am converting a message processing app that uses many COM components from VB6 to C#. Many of the COM components in the application are fine-grained components that are used in high numbers and frequency within message processing loops. I am seeing a massive (and progressively growing) increase in memory usage when processing a set of test messages in the C# app as compared to the VB6 app. I used a memory profiler on the application that confirmed that the high memory usage was due to live instances of COM objects on the application's unmanaged heap. I know that these components are not being "leaked" due to live references because if I put a GC.Collect() at the core of the message processing loop, the memory usage is flat and nearly identical to the VB6 app (although the performance degrades horribly as one would expect).
I have read everything I can find on the multi-generation garbage collector in C#, runtime callable wrappers, unmanaged resource memory allocation, Marshal.ReleaseComObject(), Marshal.FinalReleaseComObject(), etc. None of it explains why the application is holding onto live COM objects in the unmanaged heap when the corresponding RCWs are eligible for garbage collection.
In some articles, I have seen allusions to the possibility that the actual implementation of the garbage collector in C# may involve optimizations such as not performing collection of all eligible objects in a particular generation. If this or something like it were true, it could explain why eligible RCWs and their corresponding COM objects are not collected and destroyed. Another explanation could be if the destruction of a COM object in the unmanaged heap is not directly tied to the collection of its corresponding RCW. I have not found anything that offers this degree of detail on how COM objects are handled in .NET. I need to understand this better because my app's memory usage is currently unacceptable. Any pointers or recommendations would be greatly appreciated.
Edit: I should add that I am quite familiar with finalizers (which are not documented to exist on RCWs) and the IDisposable interface (which RCWs do not implement). To the best of my understanding, Marshal.ReleaseComObject() is the proper method of explicitly "disposing" of a COM reference. I was careful to add that statement for every known usage of a COM object in my app and it resulted in no difference in memory usage.
Further, it is not clear why a lack of disposal or finalization code could be the problem when the addition of an explicit GC.Collect() results in no memory problems. Neither Dispose() nor the presence of a finalizer result in the actual collection of objects. The former permits an object to suppress its finalization step (if any) and the latter allows for the cleanup of unmanaged resources of which none are exposed in an RCW.

Are you implementing IDisposable properly in the classes that instantiate your COM objects?
You need to implement IDisposable and dispose of your COM RCW's in the Dispose() method. Then all code that instantiates classes that implement IDisposable should call it either explicitly or by using a using() statement, like so:
var first = new DisposableObject();
...
first.Dispose();
and
using(var first = new DisposableObject())
{
...
}
IDisposable is the only way to get the CLR to dispose of these objects in a timely manner and to make sure you lose COM references.

Use a Finalizer or destructor to clean up the memory used by the COM objects.
http://msdn.microsoft.com/en-us/library/66x5fx1b.aspx
Alternatively, if you want the objects to clean up immediately, you can implement IDispose, and use a using statement in your code that instantiates the COM object.

Clean Up Vs Memory Reclaim in .Net

I was reading this MSDN reference:
Although the garbage collector is able
to track the lifetime of an object
that encapsulates an unmanaged
resource, it does not have specific
knowledge about how to clean up the
resource. For these types of objects,
the .NET Framework provides the
Object.Finalize method, which allows
an object to clean up its unmanaged
resources properly when the garbage
collector reclaims the memory used by
the object. By default, the Finalize
method does nothing. If you want the
garbage collector to perform cleanup
operations on your object before it
reclaims the object's memory, you must
override the Finalize method in your
class.
I understand how GC works but this give me a thought that what is actually CleanUp? Is it just reclaiming memory if it is than why it is having different name?

Beware that this is not the full story either, as finalizing only occurs when the object is garbage collected. In actual fact you should release all unmanaged resources (file handles, mutexes, unmanaged memory) as soon as possible. You should have a look at the IDisposable interface, which defines the Dispose() function.
Wherever possible your disposer should run the same method to free resources as the finalizer would, but then call GC.SuppressFinalize() to stop it from running again (in the finalizer), as there is a minor performance hit when using objects that implement finalizers.

They used a generic phrase such as "clean up" because other things may need to be done besides just reclaiming memory. I can see how this may be a little confusing, since the quote mentions cleaning up resources and reclaiming memory in the same sentence. In that case, what they mean is that the garbage collector reclaims the memory used by the managed code that actually called into an unmanaged library (a wrapper class, for example), but leaves the unmanaged-specific reclamation process up to the developer (closing file handles, freeing buffers, etc).
As an example, I have a Graphviz wrapper library containing a Graph class. This class wraps the functions used to create graphs, add nodes to them, etc. Internally, this class maintains a pointer to an unmanaged graph structure allocated by Graphiz itself. To the .NET Framework, this is merely an IntPtr and it has no idea how to free it during garbage collection. So, when a managed Graph object is no longer being used, the garbage collector frees up the memory used by the pointer, but not the data it points to. To do this, I have to implement a finalizer that calls the unmanaged function agclose (the Graphviz function that releases the resources used by a graph).

An example would be if you wrote a component that used some operating system resource like named pipe or memory mapped file. You could used the finalize operation to release the resource back to the os.

Cleaning up a non-managed resource could include closing network connections, files, database connections, etc.. Of course, it could also include the deallocation of memory for that resource.

CleanUp is here means free up any bounded resource (Harddisk, Network bandwidth, Sound Card, Memory, CPU, etc) and since .NET have no managed reference to unmanaged code, it could just let you do the job at the right moment yourself using Finalize() method before GC swaps it. If you don't CleanUp you would end up some orphan unmanaged code at an unknown state which are still using resources. It's better to implement IDisposable and CleanUp by calling Dispose() on your object.

Release resources in .Net C#

I'm new to C# and .NET, ,and have been reading around about it.
I need to know why and when do I need to release resources? Doesn't the garbage collector take care of everything? When do I need to implement IDisposable, and how is it different from destructor in C++?
Also, if my program is rather small i.e. a screensaver, do I need to care about releasing resources?
Thanks.

The garbage collector is only aware of memory. That's fine for memory, because one bit of memory is pretty much as good as any other, so long as you've got enough of it. (This is all modulo cache coherency etc.)
Now compare that with file handles. The operating system could have plenty of room to allocate more file handles - but if you've left a handle open to a particular file, no-one else will be able to open that particular file for writing. You should tell the system when you're done with a handle - usually by closing the relevant stream - as soon as you're finished, and do so in a way that closes it even if an exception is thrown. This is usually done with a using statement, which is like a try/finally with a call to Dispose in the finally block.
Destructors in C++ are very different from .NET finalizers, as C++ destructors are deterministic - they're automatically called when the relevant variable falls out of scope, for example. Finalizers are run by the garbage collector at some point after an object is no longer referenced by any "live" objects, but the timing is unpredictable. (In some rare cases, it may never happen.)
You should implement IDisposable yourself if you have any clean-up which should be done deterministically - typically that's the case if one of your instance variables also implements IDisposable. It's pretty rare these days to need to implement a finalizer yourself - you usually only need one if you have a direct hold on operating system handles, usually in the form of IntPtr; SafeHandle makes all of this a lot easier and frees you from having to write the finalizer yourself.

Basically, you need to worry about releasing resources to unmanaged code - anything outside the .NET framework. For example, a database connection or a file on the OS.
The garbage collector deals with managed code - code in the .NET framework.
Even a small application may need to release unmanaged resources, for example it may write to a local text file. When you have finished with the resource you need to ensure the object's Dispose method is called. The using statement simplifies the syntax:
using (TextWriter w = File.CreateText("test.txt"))
{
w.WriteLine("Test Line 1");
}
The TextWriter object implements the IDisposable interface so as soon as the using block is finished the Dispose method is called and the object can be garbage collected. The actual time of collection cannot be guaranteed.
If you create your own classes that need to be Disposed of properly you will need to implement the IDisposable interface and Dispose pattern yourself. On a simple application you probably won't need to do this, if you do this is a good resource.

Resources are of two kinds - managed, and unmanaged. Managed resources will be cleaned up by the garbage collector if you let it - that is, if you release any reference to the object. However, the garbage collection does not know how to release unmanaged resources that a managed object holds - file handles, and other OS resources for example.
IDisposable is best practice when there's a managed resource you want released promptly (like a database connection), and vital when there are unmanaged resources which you need to have released. The typical pattern:
public void Dispose()
protected void Dispose(bool disposing)
Lets you ensure that unmanaged resources are released whether by the Dispose method or by object finalisation.

You don't need to release memory in managed objects like strings or arrays - that is handled by the garbage collector.
You should clean up operating system resources and some unmanaged objects when you have finished using them. If you open a file you should always remember to close that file when you have finished using it. If you open a file exclusively and forget to close, the next time you try to open that file it might still be locked. If something implements IDisposable, you should definitely consider whether you need to close it properly. The documentation will usually tell you what the Dispose method does and when it should be called.
If you do forget, the garbage collector will eventually run the finalizer which should clean up the object correctly and release the unmanaged resources, but this does not happen immediately after the object becomes eligible for garbage collection, and it in fact might not run at all.
Also it is useful to know about the using statement.

The garbage collector releases MEMORY and cleans up - through disposition - elemetns it removes. BUT: IT only does so when it has memory pressure.
THis is seriously idiotic for ressources whree I may want to explicitely release them. Save to file, for example, is supposed to: Open the file, write out the data and - close the file, so it can be copied away by the user if he wants, WITHOUT waiting for the GC to come around and release the memory for the file object, which may not happen for hours.

You only need to worry about precious resources. Most objects you create while programming do not fit into this category. As you say, the garbage collector will take care of these.
What you do need to be mindful of is objects that implement IDisposable, which is an indication that the resources it owns are precious and should not wait for the finalizer thread to be cleaned up. The only time you would need to implement IDisposable is on classes that own a) objects that implement IDisposable (such as a file stream), or b) unmanaged resources.