We Keep Coding

Program not releasing handles

I have a C# program that is doing some heavy lifting. It's creating many objects and running genetic algorithms with them.
Right now I have the issue where the program, when running at full clip (i.e. no attached debuggers or profilers) is creating ~300 handles/second. What's more, it doesn't seem to be releasing any of them. After about 15/16 hours, there are no more handles available and the program dies.
The problem I've had with debugging is that my memory management seems to be pretty good based on my profiler (JustTrace). The objects seem to be getting cleaned up (though there are a lot of Gen2 objects laying around after many hours of running), and the program seems to only take up 35MB of memory at the max. It still holds on to its handles though!
If I close the program, all the handles get released just fine.
Where should I be looking? Is there something that could be holding on to an excess of handles but not be related to the objects in memory?
Edit: Note, when I say "Handles", I mean I open up Task Manager in Windows and look at "Handles" in the "System" box on the Performance tab.
Edit2: So a couple of weeks late, I find out it's my antivirus preventing the thread handles from being released. What's strange is that the antivirus we're using shouldn't be doing this kind of process scanning according to what I've read. What's more, I don't have the ability to exclude my process from the AV scanning!
Is there something I could be doing to give the AV a chance to release the threads? I've tried adding some extra sleeps and that doesn't seem to be working. AV is BitDefender Endpoint Security 2015.

I usually use JustTrace as well, but I had a similar problem recently where the profiler wasn't telling me the whole story.
The redgate ANTS profiler can also show you unmanaged memory which was the problem in my case. Looking at the unmanaged memory may give you a clue as to what's going on.

It turns out my Antivirus was preventing the thread handles from being released. I haven't found a way to prevent this from happening except disabling/switching antiviruses. In the case I find a better way, I'll update here.

Targetting memory leak in a .NET production service

I have a C#.NET service running in production. The service functions as a TCP server to which clients register and make requests against. In looking at the Task Manager, it appears to be leaking about 10MB/day. I don't seem to notice these in dev (perhaps because of far less traffic and client activity). In searching around I've read that the Task Manager can be seriously wrong, but I'm not sure how accurate this is or in what circumstances the TM would display incorrect information.
To solve this problem I need to more closely monitor memory consumption. The problem is that the leak only seems to appear in production, where the deployed service was built for Release. Also since it's a service that can't be run directly be VS with an attached profiler/debugging, I'm not sure how to best pinpoint the problem with something more precise than TM.
Any group wisdom would be much appreciated, thanks.
EDIT:
I've added perfmon counters for the privates bytes of the service (7MB to start out) as well as CLR mem in all heaps (30MB to start out)
Task manager says the total memory to be ~37MB so this seems to make sense
The first part of this is to let the service go for a day and check out my counters again.
If my private bytes get huge but CLR mem is roughly static this would indicate an unmanaged leak. If both get huge then it's a managed leak.
Thanks guys.

Your first task is figuring out if the process is leaking memory. You can do this with perfmon measuring the Private Bytes
http://www.goldstarsoftware.com/papers/CapturingVirtualBytesToALogFile.pdf
If the graph is consistently rising (for say half an hour ) you have a memory leak. You can then use other counters to figure out if this is a .NET leak (.NET memory) though this is unlikely. I find that in most of these cases, there is a COM component that is being invoked but not released.
If you truly have a memory leak (and this isn't just variable memory usage)- the process will shutdown with an out of memory exception after running for a while.

You need one of the below MemoryProfilers in order to monitor it;
http://www.jetbrains.com/profiler/
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/
There are other choices but these are very capable and you can profile remote application's memory with them (at least JetBrains's solution handles that)

Follow this guide: http://blogs.msdn.com/b/tess/archive/2008/03/25/net-debugging-demos-lab-7-memory-leak.aspx
It goes over exactly what you're describing, a memory leak in production. As was mentioned you have to first determine whether it's unmanaged code or managed code that's leaking using perfmon and Private Bytes.
In general make sure for networking objects you're wrapping them in using statements so that they're properly disposed.
A workflow I often use for managed memory leaks is to start the server on a test machine, hit it with a known amount of connections (say 123,456 connections). Then take a memory snapshot by going to task manager and right clicking on the process name and selecting 'create dump'. Open this dump with WinDBG and SOS and run the command !dumpheap -stat. Look for objects that have a multiple of 123,456 instances. Should these objects still be in memory? If not run a !gcroot on an instance of those objects to find why it's still in memory.

Get a dump of the memory when its in a leak state using the Task Manager right click on the process and select create dump file. You can also use ProcDump which gives you more options.
Use SOS Extensions in either WinDebug or Visual Studio to inspect the memory.

System.OutOfMemory being thrown. How to find the culprit?

I am using Visual C# Express 2008 and I have an application that starts up on a form, but uses a thread with a delegated display function to take care of essentially all the processing. That way my form doesn't lock up while tasks are being processed.
Semi-recently, after going through a repeated process a number of times (the program processes incoming data, so when data comes in, the process repeats) my app will crash with a System.OutOfMemory error.
The stack trace in the error message is useless because it only directs me to the the line where I call the delegated form control function.
I've heard people say they use ProcMon from SysInternals to see why errors like this happen. But I, for the life of me, can't figure it out. The amount of memory I am using doesn't change as the program runs, if it goes up, it comes back down. Plus, even if it was going up, how do I figure out which part of my program is the problem?
How can I go about investigating this problem?
EDIT:
So, after delving further into this issue, I looked through anything that I was ever re-declaring. There were a few instances where I had hugematrix = new uint[gigantic], so I got rid of about 3 of those.
Instead of getting rid of the error, it is now far more obscured and confusing.
My application takes the incoming data, and renders it using OpenGL. Now, instead of throwing "System.OutOfMemory" it simply does not render anything with OpenGL.
The only difference in my code is that I do not make new matrices for holding the data I plot. That way, I hope, my array stays in the same place in memory and doesn't do anything suicidal to my LOH.
Unfortunately, this twists the beast far beyond my meager means. With zero errors popping up, and all my data structures apparently still properly filled, how can I find my problem? Does OpenGL use memory in an obscure way so as to not throw exceptions when it fails? Is memory still a problem? How do I find out? All the memory profilers in the world seem to tell me very little.
EDIT:
With the boatloads of support from this community (with extra kudos to Amissico) the error has finally been rooted out. Apparently I was adding items to an OpenGL list, and never taking them off the list.
The app that finally clued me in was .Net Memory Profiler. At the time of crash it showed 1.5GB of data in the <unknown> category. Through process of elimination (everything else in the list that was named), the last thing to be checked off the list was the OpenGL rendering pipleline. The rest is history.

Based on the description in your comments, I would suspect that you are either not disposing of your images correctly or that you have severe Large Object Heap fragmentation and, when trying to allocate for a new image, don't have enough contiguous space available. See this question for more info - Large Object Heap Fragmentation

You need to use a memory profiler, such as the ants memory profiler to find out what causes this error.
Are you re-registering an event handler on every loop and not un-registering it?

CLR Profiler for the .NET Framework 2.0 at https://github.com/MicrosoftArchive/clrprofiler
The most common cause of memory fragmentation is excessive string creation.

Following considerations:
Make sure that threads you spawn are destroyed (aborted or function return). Too much threads can fail application, although in Task Manager used memory is not too high
Memory leaks. Yes, yes, you can cause them in .net pretty well without setting reference to nulls. This can be solved by using memory profilers like dotTrace or ANTS Memory Profiler

I had an OutOfMemoryException-problem as well:
Microsoft Visual C# 2008 Reducing number of loaded dlls
The reason was fragmentation of 2GB GB virtual address space and poster nobugz suggested Sysinternal's Vmmap utility which has been very helpful for diagnostics. You can use it to check if your free memory areas become more fragmented over time. (First sort by size then by type -> refresh repeat sorting and you can see if contiguous free memory blocks become smaller)

How to detect where a Memory Leak is?

I have a large website that seems to be sucking up all the memory that is being allocated. There is nothing else on the server beside this site. Within a week it eats away the 2 gigs and requires a restart. Currently this is server 2008 32 bit using IIS 7. We are reinstalling to use 64bit and add more memory. It would be nice to be able to track down where the leaks are occurring.
So what is the best practice to tracking memory leaks?

Memory leaks in .NET are not that common, but when they happen it is most commonly due to unattached event handlers. Make sure you detach handlers, before the listeners go out of scope.
Another option is if you forget to call Dispose() on IDisposable resources. This may prevent cleanup of unmanaged resources (which are not handled by GC).
And yet another possible reason is a deadlocked finalizer. That will prevent all remaining objects in the finalizer queue from being collected.
I use WinDbg + Sos to track down leaks. The steps are as follows
Dump the heap and look for suspects
Use !gcroot to find out what is keeping the suspects alive
Repeat as necessary
Be aware that large memory usage may also be due to heap fragmentation. The regular heaps are compacted, but pinned objects can cause fragmentation. Also, the LOH is not compacted so fragmentation is not uncommon for LOH.
Excellent tutorials on WinDbg + sos here: http://blogs.msdn.com/tess/

Run a profiler on your code.
Here are two good options:
RedGate's memory profiler
Jetbrains dotTrace
I believe both products have a free trial.

Run, don't walk, over to Tess
Ferrandez's blog, If broken it is,
fix it you should, which has well
scripted labs dedicated to learning
how to diagnose and debug crash, hang
and memory issues with .NET code. She
has some of the best material I've
found to date to help get you started.
Commercial memory profilers such as ANTS and SciTech are excellent resources that will show what objects are in the heap, and how they are rooted. Most commercial memory profilers have the ability to load a memory 'snap' of a process (say from your production environment).
You can capture a memory 'snap' (see Snap v. Dump) using adplus.vbs or DebugDiag. Adplus is available as part of the Debugging Tools for Windows. DebugDiag will also have some rudimentary analysis (but seems to be more reliable on unmanaged code) automagically.
Monitor the Application
For an idea on what to monitor, see Improving .NET Performance and Scalability, specifically Chapter 15.
As to how to monitor, there are commercial tools available for that as well, however, every Windows machine also comes with Perfmon.exe, which can be used to record relevant performance counters.
Test the Application
For an idea on how to perform load, or stress, tests, check out the Patterns and Practices Performance Testing Guidance for Web Applications.
Debug the Application
Once you've identified you've got a problem (monitoring) and your able to reproduce the problem (testing) you can get down to debugging the problem. See the links for Tess - that information will carry you a long way.
Then rinse and repeat! :)
Good luck!
Z

In performance monitor, add counters for Process/Private Bytes and .NET CLR Memory/# Bytes in All Heaps. Private bytes is all memory and CLR memory is just managed. So if the CLR memory remains fairly even but the private bytes continues to grow over time, this means that the leak is in unmanaged resource. That usually means that you are not disposing of native resources properly. A good thing to look at is stuff like COM or IO (streams and files). Make sure all of that stuff gets disposed when you are done with it.

You can try using profilers such as dotTrace -- set it into a memory trace and run your application.
This should give you clues in which assemblies and areas of the application that eat up too much memory on the get go.

This is probably prevention rather then detection, but at the c# code level, you should check that any classes that use large resources such as images and other files are correctly implementing the dispose pattern. If needed you may also need to override the finalizer.
MSDN has good guidance on this subject.
If you have any classes in your application that you know make use of large resources, these are the first places to look for memory issues.

I've found that the EQATEC Profiler is pretty good, plus it's free!

Check out the Memory and Memory Leak labs on this blog post:
.NET Debugging Demos
They may be of some help. Basically, you can use WinDBG to analyze a memory dump and help determine what is eating up all of your memory.
We used a similar approach to determine that Regex was chewing up all of our memory, but only when the product was run on 64-bit machines. The learning curve is kind of steep, but WinDBG is a very powerful tool.

This new article of mine maybe useful: How to detect and avoid memory and resources leaks in .NET applications

Do you do any Office interop? If so, make sure you clean up your application objects. That's one possible culprit.
Another is global objects, such as anything that's static.

Do you have lots of dynamic pages on your site?
You can also try IBM's Purify
I suggest you try with a small set of dynamic pages disabling all others for the meantime. I hate to say this but it's very much possible that IIS 7 may also have leaks.

Look at this article on detecting .NET application memory leaks and related articles mentioned at the bottom of the page and hopefully you will find a solution or at least an idea to resolve it.
Thanks

What strategies and tools are useful for finding memory leaks in .NET?

I wrote C++ for 10 years. I encountered memory problems, but they could be fixed with a reasonable amount of effort.
For the last couple of years I've been writing C#. I find I still get lots of memory problems. They're difficult to diagnose and fix due to the non-determinancy, and because the C# philosophy is that you shouldn't have to worry about such things when you very definitely do.
One particular problem I find is that I have to explicitly dispose and cleanup everything in code. If I don't, then the memory profilers don't really help because there is so much chaff floating about you can't find a leak within all the data they're trying to show you. I wonder if I've got the wrong idea, or if the tool I've got isn't the best.
What kind of strategies and tools are useful for tackling memory leaks in .NET?

I use Scitech's MemProfiler when I suspect a memory leak.
So far, I have found it to be very reliable and powerful. It has saved my bacon on at least one occasion.
The GC works very well in .NET IMO, but just like any other language or platform, if you write bad code, bad things happen.

Just for the forgetting-to-dispose problem, try the solution described in this blog post. Here's the essence:
public void Dispose ()
{
// Dispose logic here ...
// It's a bad error if someone forgets to call Dispose,
// so in Debug builds, we put a finalizer in to detect
// the error. If Dispose is called, we suppress the
// finalizer.
#if DEBUG
GC.SuppressFinalize(this);
#endif
}
#if DEBUG
~TimedLock()
{
// If this finalizer runs, someone somewhere failed to
// call Dispose, which means we've failed to leave
// a monitor!
System.Diagnostics.Debug.Fail("Undisposed lock");
}
#endif

We've used Ants Profiler Pro by Red Gate software in our project. It works really well for all .NET language-based applications.
We found that the .NET Garbage Collector is very "safe" in its cleaning up of in-memory objects (as it should be). It would keep objects around just because we might be using it sometime in the future. This meant we needed to be more careful about the number of objects that we inflated in memory. In the end, we converted all of our data objects over to an "inflate on-demand" (just before a field is requested) in order to reduce memory overhead and increase performance.
EDIT: Here's a further explanation of what I mean by "inflate on demand." In our object model of our database we use Properties of a parent object to expose the child object(s). For example if we had some record that referenced some other "detail" or "lookup" record on a one-to-one basis we would structure it like this:
class ParentObject
Private mRelatedObject as New CRelatedObject
public Readonly property RelatedObject() as CRelatedObject
get
mRelatedObject.getWithID(RelatedObjectID)
return mRelatedObject
end get
end property
End class
We found that the above system created some real memory and performance problems when there were a lot of records in memory. So we switched over to a system where objects were inflated only when they were requested, and database calls were done only when necessary:
class ParentObject
Private mRelatedObject as CRelatedObject
Public ReadOnly Property RelatedObject() as CRelatedObject
Get
If mRelatedObject is Nothing
mRelatedObject = New CRelatedObject
End If
If mRelatedObject.isEmptyObject
mRelatedObject.getWithID(RelatedObjectID)
End If
return mRelatedObject
end get
end Property
end class
This turned out to be much more efficient because objects were kept out of memory until they were needed (the Get method was accessed). It provided a very large performance boost in limiting database hits and a huge gain on memory space.

You still need to worry about memory when you are writing managed code unless your application is trivial. I will suggest two things: first, read CLR via C# because it will help you understand memory management in .NET. Second, learn to use a tool like CLRProfiler (Microsoft). This can give you an idea of what is causing your memory leak (e.g. you can take a look at your large object heap fragmentation)

Are you using unmanaged code? If you are not using unmanaged code, according to Microsoft, memory leaks in the traditional sense are not possible.
Memory used by an application may not be released however, so an application's memory allocation may grow throughout the life of the application.
From How to identify memory leaks in the common language runtime at Microsoft.com
A memory leak can occur in a .NET
Framework application when you use
unmanaged code as part of the
application. This unmanaged code can
leak memory, and the .NET Framework
runtime cannot address that problem.
Additionally, a project may only
appear to have a memory leak. This
condition can occur if many large
objects (such as DataTable objects)
are declared and then added to a
collection (such as a DataSet). The
resources that these objects own may
never be released, and the resources
are left alive for the whole run of
the program. This appears to be a
leak, but actually it is just a
symptom of the way that memory is
being allocated in the program.
For dealing with this type of issue, you can implement IDisposable. If you want to see some of the strategies for dealing with memory management, I would suggest searching for IDisposable, XNA, memory management as game developers need to have more predictable garbage collection and so must force the GC to do its thing.
One common mistake is to not remove event handlers that subscribe to an object. An event handler subscription will prevent an object from being recycled. Also, take a look at the using statement which allows you to create a limited scope for a resource's lifetime.

This blog has some really wonderful walkthroughs using windbg and other tools to track down memory leaks of all types. Excellent reading to develop your skills.

I just had a memory leak in a windows service, that I fixed.
First, I tried MemProfiler. I found it really hard to use and not at all user friendly.
Then, I used JustTrace which is easier to use and gives you more details about the objects that are not disposed correctly.
It allowed me to solve the memory leak really easily.

If the leaks you are observing are due to a runaway cache implementation, this is a scenario where you might want to consider the use of WeakReference. This could help to ensure that memory is released when necessary.
However, IMHO it would be better to consider a bespoke solution - only you really know how long you need to keep the objects around, so designing appropriate housekeeping code for your situation is usually the best approach.

I prefer dotmemory from Jetbrains

Big guns - Debugging Tools for Windows
This is an amazing collection of tools. You can analyze both managed and unmanaged heaps with it and you can do it offline. This was very handy for debugging one of our ASP.NET applications that kept recycling due to memory overuse. I only had to create a full memory dump of living process running on production server, all analysis was done offline in WinDbg. (It turned out some developer was overusing in-memory Session storage.)
"If broken it is..." blog has very useful articles on the subject.

After one of my fixes for managed application I had the same thing, like how to verify that my application will not have the same memory leak after my next change, so I've wrote something like Object Release Verification framework, please take a look on the NuGet package ObjectReleaseVerification. You can find a sample here https://github.com/outcoldman/OutcoldSolutions-ObjectReleaseVerification-Sample, and information about this sample http://outcoldman.com/en/blog/show/322

The best thing to keep in mind is to keep track of the references to your objects. It is very easy to end up with hanging references to objects that you don't care about anymore.
If you are not going to use something anymore, get rid of it.
Get used to using a cache provider with sliding expirations, so that if something isn't referenced for a desired time window it is dereferenced and cleaned up. But if it is being accessed a lot it will say in memory.

One of the best tools is using the Debugging Tools for Windows, and taking a memory dump of the process using adplus, then use windbg and the sos plugin to analyze the process memory, threads, and call stacks.
You can use this method for identifying problems on servers too, after installing the tools, share the directory, then connect to the share from the server using (net use) and either take a crash or hang dump of the process.
Then analyze offline.

From Visual Studio 2015 consider to use out of the box Memory Usage diagnostic tool to collect and analyze memory usage data.
The Memory Usage tool lets you take one or more snapshots of the managed and native memory heap to help understand the memory usage impact of object types.

one of the best tools I used its DotMemory.you can use this tool as an extension in VS.after run your app you can analyze every part of memory(by Object, NameSpace, etc) that your app use and take some snapshot of that, Compare it with other SnapShots.
DotMemory