Sometimes you need to use a particular third-party library, like in my case, one that loads up PowerPoints and allows the user to modify them in code. We discovered that this particular library has some memory leaks, but we would still like to use it because these leaks only occur in one particular scenario that occurs very rarely. You can see objects lying around despite trying to dispose all references to them, and despite having these objects go out of scope, and despite having manually invoked garbage collection. For sure, this library creates leaks. Our application is single-threaded.
Now, that being said, I am wondering if there is any way to clean up all memory that the library has used during runtime. Are there any ways to unload and reload the DLL that might cause all memory allocations from that library to be cleared, or anything that we can do at runtime at all to clean the memory that this third party library uses and then potentially reload the library in our application?
You could investigate loading the referenced library inside a custom AppDomain; an app-domain is a unit of isolation inside a process - and can be unloaded. It does, however, require you to do some communications between the two app-domains (the default domain and the hosted domain); MarshalByRefObject is the easiest trick there.
Alternatively: just use an entire separate process for this work. On windows, creating a process is relatively expensive, but not so expensive that you should never do it. Shutting down the process when done is the equivalent of nuking it from orbit. You can always re-spawn another process later.
There is an entire series of "How to" topics about AppDomain linked from MSDN here
Related
What is the most important use of AppDomains in C#?
The single most important use is that your code has to have one - i.e. everything you write in C# executes in an AppDomain. That is quite important ;-p
If you mean additional app-domains:
When using plugins and other untrusted code, it allows you both isolation, and the ability to unload them (you can't unload assemblies - only entire app-domains).
I'm using it currently to load dynamically generated dlls, so that I can unload them.
They also allow you to set different configuration files, trust levels, etc - but have associated costs of complexity and remoting.
MSDN has a section on app-domains, here.
I can't tell you what the most important use is, since that depends on the situation.
AppDomains are useful for sandboxing parts of your application. You can load extensions in an AppDomain and unload them again - something you cannot otherwise do. You can assign specific rights to AppDomains. Per default objects in different AppDomains cannot access each other.
AppDomains can be viewed as lightweight processes as they give you many of the same features. However, unlike a Process new AppDomains do not have their own thread per default. You have to manage AppDomains and threads yourself.
Also, AppDomains all share the same managed heap. This is usually not a problem, but it may have surprising effects as some instances like strings are shared among AppDomains. For regular use this is not an issue, but if you use strings for locking, threads in different AppDomain can affect each other.
In general, it's not so daily coding practice to use AppDomains, this could be considered something as an advanced concept.. but, starting from this simple thing, it's important to better understand concepts behind the word "AppDomain".
In terms of architecture, and taking it simple as possible, an AppDomain is an isolation container even in terms of memory addressing, inside it all the assemblies needed by an application are loaded and executed, even if this concept is more complicated to explain in details (I hope it's not about yor question to going so deeper).
Starting from there, the AppDomain class first of all is used to obtain access to the application related executing application domain, this could be done via the Singleton property implementation AppDomain.CurrentDomain. In this way it's possible to:
obtain access the loaded assemblies;
obtain access to the appdomain-shared data slots;
intems marshalling, in terms of unwrapping created instances from loaded assemblies in created domains.
Then, the AppDomain class is used to:
create more "domains" in the same process;
executing assemblies in the process;
manage the appdomain's loading/unloading process.
It could be useful to take a view of the code of the new Microsoft framework (not yet released) MEF (Managed Extesibility Framework) which is truly based on concepts like AppDomains creations and unload, dynamically loaded assemblies.
As a simple example of things and examples of what you can do with AppDomains, I can share this link.
I hope I answered your question.
A C# AppDomain is a logically isolated container inside which .NET code run. When you run any .NET code it always runs in a default appdomain.
Do watch this 30 minutes youtube video What is C# AppDomain ? which explains AppDomain in more detail.
But let me still try to explain in more detail. Lets say you get a third party DLL and you want to use it in your application. But you also suspect that the third party can have some malicious code so you would like to run the third party DLL in a constrained environment. Like you do not want the third party to access your c: drive or delete files and so on.
So you can create two AppDomains one which is for the third party and one for your own C# classes. For the third party appdomain you will apply security constraint that it can not access c: drive and for your C# DLLs you will have a unrestricted app domain.
Please read my blog for standard application of runtime loading of DLLs and cross-communication using AppDomain. https://blog.vcillusion.co.in/sending-events-through-application-domain-boundary/
Runtime Loading and unloading of DLLs: I worked on a project where DLLs are loaded at runtime by the user, and during program execution, the methods are executed using Reflection and unloaded during the program run.
Securing my Main Execution Program: We are loading the DLL dynamically so any exception that occurred in that dynamically loaded DLL didn't affect my main AppDomain. In case of corruption scenarios, we have the option of efficiently unloading and loading the DLL again.
Cross-AppDomain Communication: We can dynamically load any two DLLs at runtime in different AppDomain and make them communicate with each other.
If we create a singleton class in a dll and use it in another dll which instantiates this singleton class, and an exe uses this dll at a point of time, will these two dlls get unloaded immediately after their job or will they get unloaded only when the main exe exits?
Disclaimer: I am not an expert on memory or processes/app domains, so I may have some specifics wrong, but I firmly believe that my conclusion is valid - feel free to comment and correct any specific errors.
No, the DLL is not unloaded until the app domain that contains it (typically the main process for windows/console apps and the web site for ASP.NET apps) is closed.
From MSDN:
There is no way to unload an individual assembly without unloading all of the application domains that contain it.
So you can manually create (and destroy) app domains, which will unload any DLLs, but then you have to marshal data across the domain boundaries, which may be worse than just keeping the DLL in memory.
The objects created by the DLL may be removed from memory (if nothing holds a reference to them), but the code itself will remain in memory. In the case of a pure singleton, it will stay in memory until the app closes (since the class that manages it will always have a reference). You could remove the reference to the singleton from the factory class, but you still may have other code that references it, so you risk having multiple instances of your "singleton". If that's not a concern, that may be a valid strategy to dispose of the singleton when you're done with it.
Think about most apps - they respond to human interaction in some way. Users can click any button, select any menu item, etc. in any order that the app allows. How is an application supposed to know for certain that code within a DLL will never get called again? Assemblies are relatively expensive to load (in part since they must be JITted), so it would be inefficient to unload them if there's any chance that they'll get loaded again.
Bottom line - since the memory footprint of a DLL is usually relatively small (since it just contains code and metadata) and is only loaded once for an entire domain, unless you experience significant memory pressure from it that can't me relived by managing other memory better, I wouldn't go to the trouble of trying to unload it.
I need to use an external assembly that I cannot modify. Suppose that I use a class from that assembly like so:
using (ExternalWidget widget = new ExternalWidget())
{
widget.DoSomething();
}
Each time I call this code, it leaks unmanaged memory. ExternalWidget implements IDisposable and I have wrapped it in a using statement, but ExternalWidget does not clean up its unmanaged resources.
Since I do not have access to the ExternalWidget code, I cannot fix this issue the right way. Is there any other way that I can free up the memory resources used by ExternalWidget?
If it is a true unmanaged memory leak and you can't change the code, then there's little that you can do. The framework can't pick up on that, nor can it clean up that code.
The approach in this case would be isolation of that component. It means there's going to be a lot of overhead in accessing it, but there's nothing else you can do.
You can't run the code in another application domain, because the unmanaged code has no concept of the application domain.
That leaves the process level. I'd recommend creating a service contract in WCF that mimics the calls to the ExternalWidget along with a Shutdown method.
Then, you would create an EXE which would expose this contract (with a session, so you can hold onto the ExternalWidget instance, unless each call is stateless) through named pipe bindings.
As a parameter to the EXE, it would take a unique identifier (use a Guid) and use that as part of setting up the endpoint for the WCF service.
Then, you would make the calls, and when you're done with that instance of ExternalWidget, call Shutdown; the EXE would know to stop waiting and then the process will exit, and the operating system will reclaim the memory.
Of course, there's a huge amount of overhead here, so if you find you're making a lot of calls and don't need a new process for each set of calls, you can expand the idea into a service which counts the calls and then recycles the process (the service will still have to shell out, or it will run out of resources) when needed.
Note that if this turns out to be a managed memory issue, then you can always spin up a new application domain, run your code there (marshaling the results back and forth as needed) and then release the application domain.
We are using EWS Managed API which polls MS Exchange for new mail messages after a given interval. With each invocation of the polling call (PullSubscription.GetEvents()) - Microsofts API is failing to properly dispose the NetworkStream and causes memory to proportionately increase. This was previously discussed here, but never resolved. Using ANTS Profiler we were able to determine which objects were continuously growing in memory and isolate the issue.
Now that the issue has been isolated - is there a way to dispose of a NetworkStream created in an external API that we don't have a reference to? GC.Collect() doesn't seem to dispose it since it still has an active reference. What can we do to cleanup the dangling reference? Is there some wrapper we can use to force cleanup of their buggy SDK?
There is no way to force GC to release memory for a referenced object!
First of all I would suggest to contact microsoft itself for help with this bug.
Second, are you talking about "disposal" or just memory release? They are two totally different things. (IDisposable pattern, finalizers).
Third, can u just dereference the object that are referencing these objects?
Fourth, one possible solution can be to decompile with reflector the code that is giving you the issue, understand a way you can arrive to the fields that are keeping the referenced objects, use reflection in your code to access the private fields and put them to null.
Is a very dirty hack, but if you have no other way is the only thing i can think of. Do this only if you cannot go in any other ways.
The most easy way would be running the part the is interfacing the SDK into its own AppDomain and after your are done unload the AppDomain. This will cause all the memory allocated in the AppDomain to be freed.
But you will need add some work to your project since you can only interchange with an AppDomain the a MarshalByRef object or marked as serilizable.
This will also allow you to monitor the amount of memory consumed by the AppDomain. So you can create your AppDomain, run the buggy SDK in it and if it reaches a special limit of memory consumption you can unload it.
What is the most important use of AppDomains in C#?
The single most important use is that your code has to have one - i.e. everything you write in C# executes in an AppDomain. That is quite important ;-p
If you mean additional app-domains:
When using plugins and other untrusted code, it allows you both isolation, and the ability to unload them (you can't unload assemblies - only entire app-domains).
I'm using it currently to load dynamically generated dlls, so that I can unload them.
They also allow you to set different configuration files, trust levels, etc - but have associated costs of complexity and remoting.
MSDN has a section on app-domains, here.
I can't tell you what the most important use is, since that depends on the situation.
AppDomains are useful for sandboxing parts of your application. You can load extensions in an AppDomain and unload them again - something you cannot otherwise do. You can assign specific rights to AppDomains. Per default objects in different AppDomains cannot access each other.
AppDomains can be viewed as lightweight processes as they give you many of the same features. However, unlike a Process new AppDomains do not have their own thread per default. You have to manage AppDomains and threads yourself.
Also, AppDomains all share the same managed heap. This is usually not a problem, but it may have surprising effects as some instances like strings are shared among AppDomains. For regular use this is not an issue, but if you use strings for locking, threads in different AppDomain can affect each other.
In general, it's not so daily coding practice to use AppDomains, this could be considered something as an advanced concept.. but, starting from this simple thing, it's important to better understand concepts behind the word "AppDomain".
In terms of architecture, and taking it simple as possible, an AppDomain is an isolation container even in terms of memory addressing, inside it all the assemblies needed by an application are loaded and executed, even if this concept is more complicated to explain in details (I hope it's not about yor question to going so deeper).
Starting from there, the AppDomain class first of all is used to obtain access to the application related executing application domain, this could be done via the Singleton property implementation AppDomain.CurrentDomain. In this way it's possible to:
obtain access the loaded assemblies;
obtain access to the appdomain-shared data slots;
intems marshalling, in terms of unwrapping created instances from loaded assemblies in created domains.
Then, the AppDomain class is used to:
create more "domains" in the same process;
executing assemblies in the process;
manage the appdomain's loading/unloading process.
It could be useful to take a view of the code of the new Microsoft framework (not yet released) MEF (Managed Extesibility Framework) which is truly based on concepts like AppDomains creations and unload, dynamically loaded assemblies.
As a simple example of things and examples of what you can do with AppDomains, I can share this link.
I hope I answered your question.
A C# AppDomain is a logically isolated container inside which .NET code run. When you run any .NET code it always runs in a default appdomain.
Do watch this 30 minutes youtube video What is C# AppDomain ? which explains AppDomain in more detail.
But let me still try to explain in more detail. Lets say you get a third party DLL and you want to use it in your application. But you also suspect that the third party can have some malicious code so you would like to run the third party DLL in a constrained environment. Like you do not want the third party to access your c: drive or delete files and so on.
So you can create two AppDomains one which is for the third party and one for your own C# classes. For the third party appdomain you will apply security constraint that it can not access c: drive and for your C# DLLs you will have a unrestricted app domain.
Please read my blog for standard application of runtime loading of DLLs and cross-communication using AppDomain. https://blog.vcillusion.co.in/sending-events-through-application-domain-boundary/
Runtime Loading and unloading of DLLs: I worked on a project where DLLs are loaded at runtime by the user, and during program execution, the methods are executed using Reflection and unloaded during the program run.
Securing my Main Execution Program: We are loading the DLL dynamically so any exception that occurred in that dynamically loaded DLL didn't affect my main AppDomain. In case of corruption scenarios, we have the option of efficiently unloading and loading the DLL again.
Cross-AppDomain Communication: We can dynamically load any two DLLs at runtime in different AppDomain and make them communicate with each other.