We are using EWS Managed API which polls MS Exchange for new mail messages after a given interval. With each invocation of the polling call (PullSubscription.GetEvents()) - Microsofts API is failing to properly dispose the NetworkStream and causes memory to proportionately increase. This was previously discussed here, but never resolved. Using ANTS Profiler we were able to determine which objects were continuously growing in memory and isolate the issue.
Now that the issue has been isolated - is there a way to dispose of a NetworkStream created in an external API that we don't have a reference to? GC.Collect() doesn't seem to dispose it since it still has an active reference. What can we do to cleanup the dangling reference? Is there some wrapper we can use to force cleanup of their buggy SDK?
There is no way to force GC to release memory for a referenced object!
First of all I would suggest to contact microsoft itself for help with this bug.
Second, are you talking about "disposal" or just memory release? They are two totally different things. (IDisposable pattern, finalizers).
Third, can u just dereference the object that are referencing these objects?
Fourth, one possible solution can be to decompile with reflector the code that is giving you the issue, understand a way you can arrive to the fields that are keeping the referenced objects, use reflection in your code to access the private fields and put them to null.
Is a very dirty hack, but if you have no other way is the only thing i can think of. Do this only if you cannot go in any other ways.
The most easy way would be running the part the is interfacing the SDK into its own AppDomain and after your are done unload the AppDomain. This will cause all the memory allocated in the AppDomain to be freed.
But you will need add some work to your project since you can only interchange with an AppDomain the a MarshalByRef object or marked as serilizable.
This will also allow you to monitor the amount of memory consumed by the AppDomain. So you can create your AppDomain, run the buggy SDK in it and if it reaches a special limit of memory consumption you can unload it.
I know all the theories :
I know what is GC, when to call dispose, Finalise when it is getting called.
I would like to know, in your live project ..in which scenario have used all this.
I mean when the project
manager/client insisted you to
cleanup the memory ? When you find
any errors in the programs? Kind of
error messages or error logs? When
your program got crashed because of
unwanted memory? or any other
scenarios?
You should not care when and how GC is called. It is inteligent enough to know when to run and what objects to free.
You should also Dispose, either manualy or using "using" all objects, that implement IDisposable. You will then prevent many errors with un-managed resources, like files.
And if you are running out of memory, then there is something wrong with your algorithm or code itself. Manualy calling GC.Collect is heavily discuraged, especialy in production code.
As a rule of thumb, you need to implement IDisposable if you aggregate a disposable object, or if you hold on to an unmanaged resource. Cleanup is done differently for these two scenarios. Otherwise, keep it simple, don't litter your code with dotnetisms
Similar question here
Memory leaks cause crashed up servers. Cause and effect.
Ressource management is something you just havce to do. It is not 'my client insisted I free my memory'. It is simply good practice. Not all applications may crash and the user just restarts them - there is one or other server application out there.
If you start building your programming libraries, ressource management and concurrency should be your top priority, otherwise you will never be up to speed implementing any solution.
hth
Mario
I have an application that was originally written in VB6 that I used a tool to convert to C# with pretty good success from a functional perspective. It processes a high volume of message using lots of small to medium sized COM (C++) objects.
I noticed that a particular test run in the old VB6 app that ran using less than 40M of memory required nearly 900M in the C# app. If I put a GC.Collect() in the inner-most message processing loop of the C# app, it uses the same or less memory as the VB6 app although it is then really, really slow. This leads me to believe there is no "leak" in the absolute sense of the word.
I then ran the C# app through the AQTime memory profiler and it reported that there were an excessive number of COM/C++ objects live on the heap. I hypothesized that this was because the runtime callable wrappers around the COM objects were quite small and never (or rarely) triggered collection in C# even if their referenced COM objects were substantially larger. I thought I could address this by adding explicit Marshal.ReleaseComObject() calls around the COM objects in the C# app. I went and did this in a lot of places where the lifetime of the COM objects was easy to determine. I noticed only a very slight reduction in memory usage.
I am wondering why I did not have better success with this. Looking through the static methods in the Marshal class, I see some that lead me to believe either tha I may be missing some subtlety in the handling of COM references or that my assumption that they are immediately destroyed when the RCW's reference count reaches zero is incorrect.
I would appreciate any suggestions for other approachs that I could try or other things that I may have overlooked or misunderstood.
Sorry for the link instead of a good synopsis, but I've never had that issue myself as I've dealt with IE and mshtml in a long lived scenario.
The article states:
When using a COM object from a .NET-based application, there are two objects involved: the RCW and the COM object (or objects). Garbage collection is only aware of the size of the RCW (which can be small), not of the COM object (which may be large). Therefore, while the .NET-based application might release the RCW, garbage collection may not reclaim the RCW even as memory runs out. As long as the RCW stays in memory, the COM object that it manages stays in memory also.
There are two mechanisms that ensure that COM objects are released from memory: the AppDomain object and the ReleaseComObject method. Using an AppDomain provides the simplest solution to managing COM objects but has performance costs and can expose a security risk. Using ReleaseComObject avoids those costs but requires more careful planning and coding.
COM Handling
Marshal.ReleaseComObject
Questions needing answers :
Does the finalizer of the client side ServicedComponent call ServicedComponent.DisposeObject or Dispose?
How should destruction (release of memory) occur in the com server in relation to its usage in the client?
Basically - we are reaching a 2 gig limit on process size (memory) of the COM server as memory is not being released - is the solution to call explicitly call Dispose or use the using statement in the client?
You should definitely be calling either Dispose or ServicedComponent.DisposeObject on the client side. Microsoft says "It is preferable to use the Dispose design pattern rather than DisposeObject."
Exactly what is happening in COM+ will depend on a few things:
Is the application a Library or Server application
Is object pooling being used
Is JIT being used
If the calls are out of process then it is possible that lease times may come into play. Understanding Enterprise Services (COM+) in .NET is one of the best COM+ articles I've read but, since it was written in 2002, I wonder if it is still 100% accurate.
I wrote C++ for 10 years. I encountered memory problems, but they could be fixed with a reasonable amount of effort.
For the last couple of years I've been writing C#. I find I still get lots of memory problems. They're difficult to diagnose and fix due to the non-determinancy, and because the C# philosophy is that you shouldn't have to worry about such things when you very definitely do.
One particular problem I find is that I have to explicitly dispose and cleanup everything in code. If I don't, then the memory profilers don't really help because there is so much chaff floating about you can't find a leak within all the data they're trying to show you. I wonder if I've got the wrong idea, or if the tool I've got isn't the best.
What kind of strategies and tools are useful for tackling memory leaks in .NET?
I use Scitech's MemProfiler when I suspect a memory leak.
So far, I have found it to be very reliable and powerful. It has saved my bacon on at least one occasion.
The GC works very well in .NET IMO, but just like any other language or platform, if you write bad code, bad things happen.
Just for the forgetting-to-dispose problem, try the solution described in this blog post. Here's the essence:
public void Dispose ()
{
// Dispose logic here ...
// It's a bad error if someone forgets to call Dispose,
// so in Debug builds, we put a finalizer in to detect
// the error. If Dispose is called, we suppress the
// finalizer.
#if DEBUG
GC.SuppressFinalize(this);
#endif
}
#if DEBUG
~TimedLock()
{
// If this finalizer runs, someone somewhere failed to
// call Dispose, which means we've failed to leave
// a monitor!
System.Diagnostics.Debug.Fail("Undisposed lock");
}
#endif
We've used Ants Profiler Pro by Red Gate software in our project. It works really well for all .NET language-based applications.
We found that the .NET Garbage Collector is very "safe" in its cleaning up of in-memory objects (as it should be). It would keep objects around just because we might be using it sometime in the future. This meant we needed to be more careful about the number of objects that we inflated in memory. In the end, we converted all of our data objects over to an "inflate on-demand" (just before a field is requested) in order to reduce memory overhead and increase performance.
EDIT: Here's a further explanation of what I mean by "inflate on demand." In our object model of our database we use Properties of a parent object to expose the child object(s). For example if we had some record that referenced some other "detail" or "lookup" record on a one-to-one basis we would structure it like this:
class ParentObject
Private mRelatedObject as New CRelatedObject
public Readonly property RelatedObject() as CRelatedObject
get
mRelatedObject.getWithID(RelatedObjectID)
return mRelatedObject
end get
end property
End class
We found that the above system created some real memory and performance problems when there were a lot of records in memory. So we switched over to a system where objects were inflated only when they were requested, and database calls were done only when necessary:
class ParentObject
Private mRelatedObject as CRelatedObject
Public ReadOnly Property RelatedObject() as CRelatedObject
Get
If mRelatedObject is Nothing
mRelatedObject = New CRelatedObject
End If
If mRelatedObject.isEmptyObject
mRelatedObject.getWithID(RelatedObjectID)
End If
return mRelatedObject
end get
end Property
end class
This turned out to be much more efficient because objects were kept out of memory until they were needed (the Get method was accessed). It provided a very large performance boost in limiting database hits and a huge gain on memory space.
You still need to worry about memory when you are writing managed code unless your application is trivial. I will suggest two things: first, read CLR via C# because it will help you understand memory management in .NET. Second, learn to use a tool like CLRProfiler (Microsoft). This can give you an idea of what is causing your memory leak (e.g. you can take a look at your large object heap fragmentation)
Are you using unmanaged code? If you are not using unmanaged code, according to Microsoft, memory leaks in the traditional sense are not possible.
Memory used by an application may not be released however, so an application's memory allocation may grow throughout the life of the application.
From How to identify memory leaks in the common language runtime at Microsoft.com
A memory leak can occur in a .NET
Framework application when you use
unmanaged code as part of the
application. This unmanaged code can
leak memory, and the .NET Framework
runtime cannot address that problem.
Additionally, a project may only
appear to have a memory leak. This
condition can occur if many large
objects (such as DataTable objects)
are declared and then added to a
collection (such as a DataSet). The
resources that these objects own may
never be released, and the resources
are left alive for the whole run of
the program. This appears to be a
leak, but actually it is just a
symptom of the way that memory is
being allocated in the program.
For dealing with this type of issue, you can implement IDisposable. If you want to see some of the strategies for dealing with memory management, I would suggest searching for IDisposable, XNA, memory management as game developers need to have more predictable garbage collection and so must force the GC to do its thing.
One common mistake is to not remove event handlers that subscribe to an object. An event handler subscription will prevent an object from being recycled. Also, take a look at the using statement which allows you to create a limited scope for a resource's lifetime.
This blog has some really wonderful walkthroughs using windbg and other tools to track down memory leaks of all types. Excellent reading to develop your skills.
I just had a memory leak in a windows service, that I fixed.
First, I tried MemProfiler. I found it really hard to use and not at all user friendly.
Then, I used JustTrace which is easier to use and gives you more details about the objects that are not disposed correctly.
It allowed me to solve the memory leak really easily.
If the leaks you are observing are due to a runaway cache implementation, this is a scenario where you might want to consider the use of WeakReference. This could help to ensure that memory is released when necessary.
However, IMHO it would be better to consider a bespoke solution - only you really know how long you need to keep the objects around, so designing appropriate housekeeping code for your situation is usually the best approach.
I prefer dotmemory from Jetbrains
Big guns - Debugging Tools for Windows
This is an amazing collection of tools. You can analyze both managed and unmanaged heaps with it and you can do it offline. This was very handy for debugging one of our ASP.NET applications that kept recycling due to memory overuse. I only had to create a full memory dump of living process running on production server, all analysis was done offline in WinDbg. (It turned out some developer was overusing in-memory Session storage.)
"If broken it is..." blog has very useful articles on the subject.
After one of my fixes for managed application I had the same thing, like how to verify that my application will not have the same memory leak after my next change, so I've wrote something like Object Release Verification framework, please take a look on the NuGet package ObjectReleaseVerification. You can find a sample here https://github.com/outcoldman/OutcoldSolutions-ObjectReleaseVerification-Sample, and information about this sample http://outcoldman.com/en/blog/show/322
The best thing to keep in mind is to keep track of the references to your objects. It is very easy to end up with hanging references to objects that you don't care about anymore.
If you are not going to use something anymore, get rid of it.
Get used to using a cache provider with sliding expirations, so that if something isn't referenced for a desired time window it is dereferenced and cleaned up. But if it is being accessed a lot it will say in memory.
One of the best tools is using the Debugging Tools for Windows, and taking a memory dump of the process using adplus, then use windbg and the sos plugin to analyze the process memory, threads, and call stacks.
You can use this method for identifying problems on servers too, after installing the tools, share the directory, then connect to the share from the server using (net use) and either take a crash or hang dump of the process.
Then analyze offline.
From Visual Studio 2015 consider to use out of the box Memory Usage diagnostic tool to collect and analyze memory usage data.
The Memory Usage tool lets you take one or more snapshots of the managed and native memory heap to help understand the memory usage impact of object types.
one of the best tools I used its DotMemory.you can use this tool as an extension in VS.after run your app you can analyze every part of memory(by Object, NameSpace, etc) that your app use and take some snapshot of that, Compare it with other SnapShots.
DotMemory