We Keep Coding

Tracking Down a .NET Windows Service Memory Leak

Before installing my windows service in production, I was looking for reliable tests that I can perform to make sure my code doesn't contain memory leaks.
However, All what I can find on the net was using task manager to look at used memory or some paid memory profiler tools.
From my understanding, looking at the task manager is not really helpful and cannot confirm the memory leakage (in case, there is).
How to confirm whether there is a memory leak or not?
Is there any free tools to find the source of memory leaks?
Note: I'm using .Net Framework 4.6 and Visual Studio 2015 Community

Well you can use task manager.
GC apps can leak memory, and it will show there.
But...
Free tool - ".Net CLR profiler"
There is a free tool, and it's from Microsoft, and it's awesome. This is a must-use for all programs that leak references. Search MS' site.
Leaking references means you forget to set object references to null, or they never leave scope, and this is almost as likely to occur in Garbage collected languages as not - lists building up and not clearing, event handlers pointing to delegates, etc.
It's the GC equivalent of memory leaks and has the same result. This program tells you what references are taking up tons of memory - and you will know if it's supposed to be that way or not, and if not, you can go find them and fix the problem!
It even has a cool visualization of what objects allocate what memory (so you can track down mistakes). I believe there are youtubes of this if you need an explanation.
Wikipedia page with download links...
NOTE: You will likely have to run your app not as a service to use this. It starts first and then runs your app. You can do this with TopShelf or by just putting the guts in a dll that runs from an EXE that implments the service integrations (service host pattern).

Although managed code implies no direct memory management, you still have to manage your instances. Those instances 'claim' memory. And it is all about the usage of these instances, keeping them alive when you don't expect them to be.
Just one of many examples: wrong usage of disposable classes can result in a lot of instances claiming memory. For a windows service, a slow but steady increase of instances can eventually result in to much memory usage.
Yes, there is a tool to analyze memory leaks. It just isn't free. However you might be able to identify your problem within the 7 day trial.
I would suggest to take a loot at the .NET Memory Profiler.
It is great to analyze memory leaks during development. It uses the concept of snapshots to compare new instances, disposed instances etc. This is a great help to understand how your service uses its memory. You can then dig deeper into why new instances get created or are kept alive.
Yes, you can test to confirm whether memory leaks are introduced.
However, just out-of-the box this will not be very useful. This is because no one can anticipate what will happen during runtime. The tool can analyze your app for common issues, but this is not guaranteed.
However, you can use this tool to integrate memory consumption into your unit test framework like NUnit or MSTest.

Of course a memory profiler is the first kind of tool to try, but it will only tell you whether your instances keep increasing. You still want to know whether it is normal that they are increasing. Also, once you have established that some instances keep increasing for no good reason, (meaning, you have a leak,) you will want to know precisely which call trees lead to their allocation, so that you can troubleshoot the code that allocates them and fix it so that it does eventually release them.
Here is some of the knowledge I have collected over the years in dealing with such issues:
Test your service as a regular executable as much as possible. Trying to test the service as an actual service just makes things too complicated.
Get in the habit of explicitly undoing everything that you do at the end of the scope of that thing which you are doing. For example, if you register an observer to the event of some observee, there should should always be some point in time (the disposal of the observer or the observee?) that you de-register it. In theory, garbage collection should take care of that by collecting the entire graph of interconnected observers and observees, but in practice, if you don't kick the habit of forgetting to undo things that you do, you get memory leaks.
Use IDisposable as much as possible, and make your destructors report if someone forgot to invoke Dispose(). More about this method here: Mandatory disposal vs. the "Dispose-disposing" abomination Disclosure: I am the author of that article.
Have regular checkpoints in your program where you release everything that should be releasable (as if the program is performing an orderly shutdown in order to terminate) and then force a garbage collection to see whether you have any leaks.
If instances of some class appear to be leaking, use the following trick to discover the precise calling tree that caused their allocation: within the constructor of that class, allocate an exception object without throwing it, obtain the stack trace of the exception, and store it. If you discover later that this object has been leaked, you have the necessary stack trace. Just don't do this with too many objects, because allocating an exception and obtaining the stack trace from it is ridiculously slow, only Microsoft knows why.

You could try the free Memoscope memory profiler
https://github.com/fremag/MemoScope.Net
I do not agree that you can trust the Task Manager to check if you have a memory leak or not. The problem with a garbage collector is that it can decide based on heuristics to keep the memory after a memory spike and do not return it to the OS. You might have a 2 GB Commit size but 90% of them can be free.
You should use VMMAP to check during the tests what type of memory your process contains. You do not only have the managed heap, but also unmanaged heap, private bytes, stacks (thread leaks), shared files and much more which need to be tracked.
VMMap has also command line interface which makes it possible to create snapshots at regular intervals which you can examine later. If you have a memory growth you can find out which type of memory is leaked which needs depending on the leak type different debugging tooling approaches.

I would not say that the Garbage collector is infallible. There are times when it fails unknowingly and they are not so straight forward. Memory streams are a common cause of memory leaks. You can open them in one context and they may never even get closed, even though the usage is wrapped in a using statement (the definition of a disposable object that should be cleaned up immediately after its usage falls out of scope). If you are experiencing crashes due to running out of memory, Windows does create dump files that you can sift through.
enter link description here
This is by no means fun or easy and is quite tedious but it tends to be your best bet.
Common areas that are easy to create memory leaks are anything that is using the System.Drawing dll, memory streams, and if you are doing some serious multi-threading.

If you use Entity Framework and a DI pattern, perhaps using Castle Windsor, you can easily get memory leaks.
The main thing to do is use the using( ){ } statement where-ever you can to automatically mark objects as disposed.
Also, you want to turn off automatic tracking on Entity Framework where you are only reading and not writing. Best to isolate your writes, use a using() {} at this point, get a dbContext (with tracking on), write your data.
If you want to investigate what is on the heap. The best tool I've used is RedGate ANTS http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/solving-memory-problems/getting-started not cheap but it works.
However, by using the using() {} pattern where-ever you can (don't make a static or singleton DbContext and never have one context in a massive loop of updates, dispose of them as often as you can!) then you find memory isn't often an issue.
Hope this helps.

Unless you're dealing with unmanaged code, i would be so bold to say you don't have to worry about memory leaks. Any unreferenced object in managed code will be removed by the garbage collector, and the possibility in finding a memory leak within the .net framework i would say you should be considered very lucky (well, unlucky). You don't have to worry about memory leak.
However, you can still encounter ever-growing memory usage, if references to objects are never released. For example, say you keep an internal log structure, and you just keep adding entries to a log list. Then every entry still have references from the log list and therefore will never be collected.
From my experience, you can definitely use the task manager as an indicator whether your system has growing issues; if the memory usage steadily keep rising, you know you have an issue. If it grows to a point but eventually converges to a certain size, it indicates it has reached its operating threshold.
If you want a more detailed view of managed memory usage, you can download the process explorer here, developed by Microsoft. It is still quite blunt, but it gives a somewhat better statistical view than task manager.

C# Garbage Collection -> to C++ delete

I'm converting a C# project to C++ and have a question about deleting objects after use. In C# the GC of course takes care of deleting objects, but in C++ it has to be done explicitly using the delete keyword.
My question is, is it ok to just follow each object's usage throughout a method and then delete it as soon as it goes out of scope (ie method end/re-assignment)?
I know though that the GC waits for a certain size of garbage (~1MB) before deleting; does it do this because there is an overhead when using delete?
As this is a game I am creating there will potentially be lots of objects being created and deleted every second, so would it be better to keep track of pointers that go out of scope, and once that size reachs 1MB to then delete the pointers?
(as a side note: later when the game is optimised, objects will be loaded once at startup so there is not much to delete during gameplay)

Your problem is that you are using pointers in C++.
This is a fundamental problem that you must fix, then all your problems go away. As chance would have it, I got so fed up with this general trend that I created a set of presentation slides on this issue. – (CC BY, so feel free to use them).
Have a look at the slides. While they are certainly not entirely serious, the fundamental message is still true: Don’t use pointers. But more accurately, the message should read: Don’t use delete.
In your particular situation you might find yourself with a lot of long-lived small objects. This is indeed a situation which a modern GC handles quite well, and which reference-counting smart pointers (shared_ptr) handle less efficiently. If (and only if!) this becomes a performance problem, consider switching to a small object allocator library.

You should be using RAII as much as possible in C++ so you do not have to explicitly deleteanything anytime.
Once you use RAII through smart pointers and your own resource managing classes every dynamic allocation you make will exist only till there are any possible references to it, You do not have to manage any resources explicitly.

Memory management in C# and C++ is completely different. You shouldn't try to mimic the behavior of .NET's GC in C++. In .NET allocating memory is super fast (basically moving a pointer) whereas freeing it is the heavy task. In C++ allocating memory isn't that lightweight for several reasons, mainly because a large enough chunk of memory has to be found. When memory chunks of different sizes are allocated and freed many times during the execution of the program the heap can get fragmented, containing many small "holes" of free memory. In .NET this won't happen because the GC will compact the heap. Freeing memory in C++ is quite fast, though.
Best practices in .NET don't necessarily work in C++. For example, pooling and reusing objects in .NET isn't recommended most of the time, because the objects get promoted to higher generations by the GC. The GC works best for short lived objects. On the other hand, pooling objects in C++ can be very useful to avoid heap fragmentation. Also, allocating a larger chunk of memory and using placement new can work great for many smaller objects that need to be allocated and freed frequently, as it can occur in games. Read up on general memory management techniques in C++ such as RAII or placement new.
Also, I'd recommend getting the books "Effective C++" and "More effective C++".

Well, the simplest solution might be to just use garbage collection in
C++. The Boehm collector works well, for example. Still, there are
pros and cons (but porting code originally written in C# would be a
likely candidate for a case where the pros largely outweigh the cons.)
Otherwise, if you convert the code to idiomatic C++, there shouldn't be
that many dynamically allocated objects to worry about. Unlike C#, C++
has value semantics by default, and most of your short lived objects
should be simply local variables, possibly copied if they are returned,
but not allocated dynamically. In C++, dynamic allocation is normally
only used for entity objects, whose lifetime depends on external events;
e.g. a Monster is created at some random time, with a probability
depending on the game state, and is deleted at some later time, in
reaction to events which change the game state. In this case, you
delete the object when the monster ceases to be part of the game. In
C#, you probably have a dispose function, or something similar, for
such objects, since they typically have concrete actions which must be
carried out when they cease to exist—things like deregistering as
an Observer, if that's one of the patterns you're using. In C++, this
sort of thing is typically handled by the destructor, and instead of
calling dispose, you call delete the object.

Substituting a shared_ptr in every instance that you use a reference in C# would get you the closest approximation at probably the lowest effort input when converting the code.
However you specifically mention following an objects use through a method and deleteing at the end - a better approach is not to new up the object at all but simply instantiate it inline/on the stack. In fact if you take this approach even for returned objects with the new copy semantics being introduced this becomes an efficient way to deal with returned objects also - so there is no need to use pointers in almost every scenario.

There are a lot more things to take into considerations when deallocating objects than just calling delete whenever it goes out of scope. You have to make sure that you only call delete once and only call it once all pointers to that object have gone out of scope. The garbage collector in .NET handles all of that for you.
The construct that is mostly corresponding to that in C++ is tr1::shared_ptr<> which keeps a reference counter to the object and deallocates when it drops to zero. A first approach to get things running would be to make all C# references in to C++ tr1::shared_ptr<>. Then you can go into those places where it is a performance bottleneck (only after you've verified with a profile that it is an actual bottleneck) and change to more efficient memory handling.

GC feature of c++ has been discussed a lot in SO.
Try Reading through this!!
Garbage Collection in C++

Want to understand how a managed language like C# has memory leaks

Since C# is a managed language that performs garbage collection automatically to clean up objects etc, ...
what are the ways one can introduce a memory leak?
Are there some non-obvious ways that one should look out for?
How can you detect or look for memory leaks (once you understand how they are generated etc.)

Usually leaks show up in the form of a developer writing code that "holds on" to objects when they shouldn't be, which subsequently disallows the garbage collector from collecting on those objects.
The garbage collector is pretty good at what it does, but if you don't understand what it's doing, the likelihood of you introducing memory issues into your program is pretty high.
I would suggest reading up on the GC and understanding how it works.
Here's something to get you started:
http://www.simple-talk.com/dotnet/.net-framework/understanding-garbage-collection-in-.net/

Two.
First, building up referenced objects. Like creating objects that subscribe to an event on a form. The form is active, so can not be collected, and the event subscriptions... keep the objects alive.
Second, block the garbage colelctor in native code. Like the oracle ODP.NET driever does under certain conditions. Stops the finalizer, so any object requiring finalization WILL NOT GET IT - and thus never be released.

The obvious "memory leak" one could cause in a GC-ed language would be caused simply be retaining a reference to an object after it's needed - this is especially likely if you roll your own caching or keep other global state.
Another way would be leaking memory in unmanaged resources that weren't disposed of, although most of the standard library classes will probably dispose of those in destructors so the memory will be reclaimed sooner or later.
(I'm marking the post as CW because of the open-ended nature of the question.)

A memory leak means keeping in memory objects you don't need anymore. In C#, considering the GC collects unreferenced objects, it's equivalent to say keeping references to objects you don't need.
Think about uncorrectly scoped declarations, infinite recursion or iteration...

Garbage Collection on one object, C#

I need to dispose of an object so it can release everything it owns, but it doesn't implement the IDisposable so I can't use it in a using block. How can I make the garbage collector collect it?

You can force a collection with GC.Collect(). Be very careful using this, since a full collection can take some time. The best-practice is to just let the GC determine when the best time to collect is.
Does the object contain unmanaged resources but does not implement IDisposable? If so, it's a bug.
If it doesn't, it shouldn't matter if it gets released right away, the garbage collector should do the right thing.

If it "owns" anything other than memory, you need to fix the object to use IDisposable. If it's not an object you control this is something worth picking a different vendor over, because it speaks to the core of how well your vendor really understands .Net.
If it does just own memory, even a lot of it, all you have to do is make sure the object goes out of scope. Don't call GC.Collect() — it's one of those things that if you have to ask, you shouldn't do it.

You can't perform garbage collection on a single object. You could request a garbage collection by calling GC.Collect() but this will effect all objects subject to cleanup. It is also highly discouraged as it can have a negative effect on the performance of later collections.
Also, calling Dispose on an object does not clean up it's memory. It only allows the object to remove references to unmanaged resources. For example, calling Dispose on a StreamWriter closes the stream and releases the Windows file handle. The memory for the object on the managed heap does not get reclaimed until a subsequent garbage collection.
Chris Sells also discussed this on .NET Rocks. I think it was during his first appearance but the subject might have been revisited in later interviews.
http://www.dotnetrocks.com/default.aspx?showNum=10
This article by Francesco Balena is also a good reference:
When and How to Use Dispose and Finalize in C#
http://www.devx.com/dotnet/Article/33167/0/page/1

Garbage collection in .NET is non deterministic, meaning you can't really control when it happens. You can suggest, but that doesn't mean it will listen.
Tells us a little bit more about the object and why you want to do this. We can make some suggestions based off of that. Code always helps. And depending on the object, there might be a Close method or something similar. Maybe the useage is to call that. If there is no Close or Dispose type of method, you probably don't want to rely on that object, as you will probably get memory leaks if in fact it does contain resourses which will need to be released.

If the object goes out of scope and it have no external references it will be collected rather fast (likely on the next collection).

BEWARE: of f ra gm enta tion in many cases, GC.Collect() or some IDisposal is not very helpful, especially for large objects (LOH is for objects ~80kb+, performs no compaction and is subject to high levels of fragmentation for many common use cases) which will then lead to out of memory (OOM) issues even with potentially hundreds of MB free. As time marches on, things get bigger, though perhaps not this size (80 something kb) for LOH relegated objects, high degrees of parallelism exasperates this issue due simply due to more objects in less time (and likely varying in size) being instantiated/released.
Array’s are the usual suspects for this problem (it’s also often hard to identify due to non-specific exceptions and assertions from the runtime, something like “high % of large object heap fragmentation” would be swell), the prognosis for code suffering from this problem is to implement an aggressive re-use strategy.
A class in Systems.Collections.Concurrent.ObjectPool from the parallel extensions beta1 samples helps (unfortunately there is not a simple ubiquitous pattern which I have seen, like maybe some attached property/extension methods?), it is simple enough to drop in or re-implement for most projects, you assign a generator Func<> and use Get/Put helper methods to re-use your previous object’s and forgo usual garbage collection. It is usually sufficient to focus on array’s and not the individual array elements.
It would be nice if .NET 4 updated all of the .ToArray() methods everywhere to include .ToArray(T target).
Getting the hang of using SOS/windbg (.loadby sos mscoreei for CLRv4) to analyze this class of issue can help. Thinking about it, the current garbage collection system is more like garbage re-cycling (using the same physical memory again), ObjectPool is analogous to garbage re-using. If anybody remembers the 3 R’s, reducing your memory use is a good idea too, for performance sakes ;)

How to avoid Memory Leaks? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What are some tips I can use to avoid memory leaks in my applications? Are there any gotchas or pitfalls that I can look out for?

Call Dispose on IDisposable objects or use the using clause. That should take care of most of the leaks I can think of.

Watch that you remove any event handlers that you use. In .NET, they are the most common cause of leaked memory.

As mentioned by ocdecio be sure to call Dispose on Idisposable objects, and remember to remove event handlers when you're done with an object. When building classes that works with unmanaged resources, be sure to implement Idisposable, so the user will know that there are critical resources that'll need to be disposed of.
Also, even though garbage collections do quite a bit a work for you, you should get rid of references to objects that you're done with. Else they'll still have a root, and they won't be GC'ed.

Don't underestimate the helpfulness of tools in these situations. The .NET memory profilers are fairly mature and robust nowadays, and if you have a complicated application where an object that you think should be freed is still held as a reference by something else the ability to pinpoint that reference is invaluable.
I've just finished hunting down a memory leak where a WPF tab page hosted a Windows Form control (since these tabs held a lot of data and you could open and close them at will simply waiting until the GC cleared the memory on close was not an option). I used the YourKit profiler to take a snapshot of the memory before the tab was opened, opened a tab, closed it again and took another snapshot. Inside the profiler you could visually compare the two snapshots and see what objects had survived the process and follow their references back to the GC root. I have no experience with other profilers, but I know there are a few out there if YourKit doesn't fulfil your needs.
Edited to add:
Okay, this isn't avoiding memory leaks, it's fixing them. But I'll leave it here since I think it is useful information and I don't think that enough .NET developers know about these tools.

I know some people are going to advise garbage collection as the solution. But there are lots of cases where garbage collection doesn't give you the results that you expect. It is easy to end up holding on to a stray reference which prevents whole chains of memory from being freed. Read about how this torpedoed a DARPA Grand Challenge entry. You can argue these aren't memory leaks but if the program expects that memory to be freed it is still a problem. Just like in C programming, after a couple of months you get the hang of how to make sure you don't leave unwanted references behind.

Memory leaks are bugs, so in general - the question could be answered the same as "how to code without bugs"? In the long run - it is not possible to have no bugs, but you can limit the chance for having those in the released code.
Start with caring about developed code quality and following the guidelines mentioned by others.
Simplicity is golden - the more simple the code - the less a chance for bugs or leaks
Be careful when using unmanaged resources - Windows Handles, DB connections, GDI objects, etc.
Use using for IDisposables
Implement IDisposable for classes that carry unmanaged resources
Make sure to remove any references to unused objects - including the tricky event handlers
On top of these - implement tests to see if the memory leaks - unit, concurrency, stress and load tests could help here most. See if memory leaks by checking metrics (perf counters). You could also log object creations and destructions and analyze the logs at the end of a test.

I've ran into issues where an object (Ping) implemented Dispose() twice, by implementing the IDisposable interface and inheriting it at the same time. The inherited method did nothing, and as a result you had to cast the object to IDisposable when calling Dispose() or it would leak memory. Here's a post I wrote a few years ago with more detail.

Wrap anything which is disposable in a using construct.
Avoid COM objects.
Check to see that all event hanlders are being removed properly.
Check to see that all data bindings are being removed properly.
Keep it simple
If your application logic is getting needlessly complex, you might start ending up with memory leaks. If you keep your classes small and follow general coding practices you probably won't run into any memory leaks with managed code. It is possible, but not as likely as it use to be.
If you suspect a memory leak, use a profiler to see if any objects are being kept around longer than needed.
The last time I ran into a serious memory leak was .NET 1.1, it turned out there was a bug in the framework.

Having a basic understanding of how the garbage collector works will help you avoid abusing memory. For example, if you are keeping a reference to an object you no longer need, the gc won't be able to collect it.
Along the same lines, if you're storing data the the user enters, or data that is added over time, you should consider some kind of limitations so that your memory usage doesn't grow indefinitely.

Most memory leaks that I have encountered in .NET has been related to using COM objects and not releasing them properly. As soon as I see a reference to a COM object, I think "memory leak".

The most common case of memory not being destroyed by the GC is when you have event handlers that did not get unhooked properly. You can unhook in Dispose() if you want.
I describe the problem more in detail, and I have a way to write tests to determine if you leak the object here.

As others have said call Dispose() (or use a using statement), but additionally consider whether classes use resources and if so implement IDisposeable. This is most often the problem in my code is that I have a class with member that doesn't get cleaned up till a GC.

It's managed code, that c#, so you have to try hard to leak memory :P
Try google:
http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=Mbp&q=c%23+memory+leaks&btnG=Search

Types which implement a finalizer may leak, if any one finalizer blocks for some reason. I have seen finalizers block due to locking and thread apartment issues.
As instances of types with finalizers are not collected until their respective finalizers have run a single blocking finalizer will block any other finalizable objects pending collection.

First, let me bring share my strict understanding of a memory leak. My definition of a memory leak is when you have memory you have allocated and no longer a reference to it, so it is not possible to free that memory. Having said that, it is impossible to have a memory leak in .net objects (instances of CTS types that live in the managed heap, I mean). Unreferenced memory is precisely what the GC looks for to free.
Having said that, one can have a more lax understanding of what a memory leak is. If you consider a memory leak to be a uncontrolled growth of the memory being used, well, that is very easy. Just make a big misuse of static variables, mostly ones that reference huge lists. If you keep those objects referenced, the GC will never clean them, promoting them to higher generations and making them even harder to collect. Even though this is not a memory leak in the strict sense, in the end of the day it can lead to similar symptoms. A good way to try to detect this kind of "leak" is to use the CLR Profiler.
Another source of "leaks" is through improper use of event handlers as previously stated. Every time that object A registers one of its instance methods with an event at object B, object B ends keeping an indirect reference to object A, which means that while B is alive, A will be kept alive. Please note however, that there is no circularity here. As soon as neither B or A have any root reference, no matter how many cross references they have, they will eventually be collected.
Finally, one can actually induce a memory leak in .net, but never (at least theoretically) when talking about managed memory, as the GC does an excellent job in clearing that. If any of your objects maintains a reference to unmanaged memory, for instance through interop, then that memory needs to be explicitly cleaned. Failing to do so can lead to a memory leak, indeed. Even though I've never experienced such a scenario, at least in theory this can happen. As previously stated, objects that hold such references ought to implement IDiposable in order to clear that memory and their usage should guarantee Dispose is invoked for that purpose, mainly through the usage of the using clause. Notice that Dispose will not free the object's memory, but will only ask the object to release any unmanaged memory it is referring to.
One special kind of unmanaged memory is the one needed by COM objects used in interop scenarios. These objects are accessed through Runtime Callable Wrappers, RCWs for friends, but have no Dispose. "Using" will not work. The way to release the underlying COM objects is through the static method:
System.Runtime.InteropServices.Marshal.ReleaseComObject(object);
Since "using" is only syntax sugar to call IDisposable.Dispose() in a finally block, it is not usable with RCWs, hence don't forget to place the call in ReleaseComObject(object) in a finally block yourself so you ensure it is really called.

Use "using" keyword to automatically call Dispose() method of IDisposable object.
For any COM interop you have to manually release all resources.

You may find my new article useful: How to detect and avoid memory and resources leaks in .NET applications