Looks like I'm trying to do something that other's haven't run into again. Atleast, from Google searching it seems like this is pretty unique.
I have a server running with a directory full of dlls. Each dll is used to process a certain type of order and managed by different people. When an order comes in, there is a table that determines which dll to spin up, run it and then spin it down (free from memory).
The process we started with was:
Create an AppDomain
Load needed dll into the domain
Run the code we need
Unload the AppDomain
The issue we had with this was the server would run out of memory or we would get "token errors". I think the issue is that we call some 3rd party pascal dlls I don't think are getting freed.
We updated our the process to:
Create an AppDomain on process load
Load the dll into this "global" AppDomain
Run the dll's code
This got rid of our memory issues and we have recieved zero token errors. However, when we want to move a dll from our test server to our production server we have to kill the whole service, and then the dlls are not locked in memory. Then reprocess any orders that died with the service and any that came in while the service was down.
I've worked with the group and tried to unload the AppDomain, but once it is unloaded it doesn't just free dll's that are running, it basically kills the whole domain. I've considered moving this dll loader to it's own System.Process, so that the OS can clean up the memory when the process ends. However, I'm not sure how to have one Process run code in another Process. This plan seems like basic OS security features would prevent it because virus' would have a heyday.
Does anyone have an idea how we can run dll's in an AppDomain and then having them unlocked from memory so that they can be updated? (Without creating and disposing many AppDomain because they don't seem to clean everything up.) It would be great to have a memory pool that I can nuke and but .NET isn't about manual memory management.
I'm implementing an application which will load and execute 3rd party code.
While .NET Sandboxing is fine, I can't find a way to prevent code from starting new threads.
This is a problem because AFAIK we can't enumerate and abort them safely to unload the sandbox AppDomain - we have to exit the whole process.
How could I ban Thread.Start, or (better) whitelist/blacklist specific CLR APIs?
You would need to create a scripting environment rather than run compiled code. In this environment you could parse out unsupported/unwanted keywords.
http://msdn.microsoft.com/en-us/library/ms974577.aspx
http://osherove.com/blog/2004/2/17/make-your-net-application-support-scripting-a-practical-appr.html
http://www.codeproject.com/KB/library/Dotnet_Scriptor.aspx
There might be some way of limiting permissions of code running within an AppDomain, is this what you are talking about with Sandboxing?
Good example of use of AppDomain
You could potentially force unloading of an AppDomain if "bad stuff" was occuring with cpu and memory.
more recently in .net 4 I have noticed but not investigated HostProtection Permissions ...
System.Security.Permissions
hostprotectionattribute
=== EDIT ===
It looks like CLR Hosting with Security Permission being set would be the way to go. Links...
What is CLR hosting?
http://msdn.microsoft.com/en-us/library/hbzz1a9a(v=vs.90).aspx
http://msdn.microsoft.com/en-us/library/h846e9b3(v=vs.90).aspx
http://msdn.microsoft.com/en-us/library/system.security.permissions.securitypermission(v=vs.90).aspx
Application domains allow applications to be unloaded separately. My question is how unloading an apllication can crash another application. Any example?
In theory, App Domains are completely isolated from each other (even though you may run several App Domains in one process, they act as separate processes), so this shouldn't be possible.
Are you asking theoretically, or are you actually experiencing crashes?
I assume you mean AppDomain when saying Application.
Pure managed code should unload correctly (but not all finalizers might execute)
But buggy native inter-op can of course still crash the process.
I want to have an application that works as a Host to many other small applications. Each one of those applications should work as kind of plugin to this main application. I call them plugins not in the sense they add something to the main application, but because they can only work with this Host application as they depend on some of its services.
My idea was to have each of those plugins run in a different app domain. The problem seems to be that my host application should have a set of services that my plugins will want to use and from what is my understanding making data flow in and out from different app domains is not that great of a thing.
On one hand I'd like them to behave as stand-alone applications(although, as I said, they need to use lots of times the host application services), but on the other hand I'd like that if any of them crashes, my main application wouldn't suffer from it.
What is the best (.NET) approach to this kind of situation? Make them all run on the same AppDomain but each one in a different Thread? Use different AppDomains? One for each "plugin"? How would I make them communicate with the Host Application? Any other way of doing this?
Although speed is not an issue here, I wouldn't like for function calls to be that much slower than they are when we're working with just a regular .NET application.
Thanks
EDIT: Maybe I really need to use different AppDomains. From what I've been reading, loading assemblies in different AppDomains is the only way to later be able to unload them from the process.
I've implemented something along these lines using the Managed Addin Framework (MAF) in the System.Addin namespace. With MAF you package your addins as separate DLLs, which your host app can discover and launch in its app domain, in a separate domain for all of the addins, or each addin in its own domain. With shadow copy and separate domains you can even update an addin without shutting down your hostapp.
Your host app and the addins communicate through contracts that you derive from MAF interfaces. You can send objects back and forth between the host and the addins. The cotnracts provide a black-box interface between addins and the host, allowing you to change an addin's implementation unbeknownst to the host.
Addins can even communicate between themselves if the host tells them about each other. In my case a logging addin is shared by the others. This lets me drop in different loggers without touching the other addins or the host.
For my app, the addin use simple supervisor classes that in launch worker classes on their own threads that do all of the processing. Workers catch their own exceptions, which they return to their supervisor through callback methods. Supervisors can restart workers or take other action. The host controls the supervisors through a command contract, which instructs them to start and stop workers and return data.
My host app is a Windows service. The worker threads have thrown exceptions for all the usual reasons (including bugs!), but the host app has never crashed in any of our installations. Since debugging services is inconvenient, addins allow me to build test apps that use the same contracts, with added assurance that I'm testing what I deploy.
Addins can expose UI elements, too. This is very helpful to me as I need to deploy a controller app with the host service, since services do not have UIs. Each plugin includes its own controller interface. The controller app itself is very simple - it loads the addins and displays their UI elements. This allows me to ship an updated addin with an updated interface and not have to ship a new controller.
Even though the controller and the host service use the same addins, they don't step on each other; in fact, they don't even know that another app is using the same addins. The controller and the host talk to each other through a shared database, but you could also use another inter-app mechanism like MSMQ. In the next version the host will be a WCF service with addins on the backend and web services for control.
This is a bit long-winded but I wanted to give you an idea of how versatile MAF is. It's not as complex as it might first look, and you can build rock-solid apps with it.
It depends on how much trust you wish to allow the extensions. I'm working on a similar application and I've chosen to mostly trust the extension code, as this greatly simplifies things. I call into the code from a common thread (in my case, the extensions don't really 'run' in any continuous loop, but rather execute certain tasks that the main application wants to do) and catch exceptions in this thread, so as to provide helpful warnings that loaded extensions are misbehaving.
Currently there's nothing keeping these extensions from launching their own threads that could throw and crash the whole app, but this where I've had to make the trade-off between safety and complexity. My application is not mission-critical (not like a web server or database server), so I consider it an acceptable risk that a buggy extension could bring down my application. I provide safeguards to more politely cover the most common failure cases and leave it to the plugin developers (who will mostly be in-house people for now anyway) to clean up their bugs.
In regards to Unloading, yes, you can only unload the code and metadata for an assembly if you place it in an AppDomain. That said, unless you want to be loading and unloading frequently over the life of your program, the overhead associated with keeping the code in memory is not necessarily an issue. Any actual instances or resources using types from the assembly will still be cleaned up by the GC when you stop 'using' it, so the fact that it's still in memory doesn't imply a memory leak.
If your main use case is a series of plugins that you locate once at startup and then provide an option to instantiate while your app is running, I suggest investigating the real memory footprint associated with loading all of them at start-up and keeping them loaded. If you use AppDomains, there will be additional overhead there as well (for instance, memory for the proxy objects and loaded/JITed code to support AppDomain marshaling). There will also be CPU overhead associated with the marshaling and attendant serialization.
In short, I would only use AppDomains if one of the following were true:
I want to get true isolation for the purposes of code security (i.e. I need to run untrusted code in an isolated way)
My app is mission-critical and I absolutely need to make sure that if a plugin fails, it can't bring down my core app.
I need to load and unload the same plugin repeatedly, in order to support dynamic changes to the DLL. This is mainly if my app can't stop running, but I want to hot-patch plugins while it's still running.
I would not prefer AppDomains for the sole purpose of reducing possible memory footprint by allowing Unload.
This is an interisting question.
My first idea was to simply implement interfaces from your host application in your plugin applications to allow them to communicate through Reflection, but this would only allow communication and would not bring a real "sandbox-like" architecture.
My second thought was to design a service-oriented platform. The host application would be a kind of "plugin broadcaster" which would publish your plugins in a ServiceHost on a different thread. As this need to be really responsive and "no brainer configurated", the host application could communicate with the plugin through named pipes channel (NetNamedPipesBinding for WCF) which means is only communicating with localhost pipes and does not need any network configuration or knowledge at all. I think this could be a good solution to your problem.
Regards.
Imagine an untrusted application (plugin) that reads from the standard input and writes to the standard output.
How to get the output returned for the specified input by this application preventing any side effects?
For example, if application deletes file on a disk, it should be detected and this attempt should be canceled.
It's some kind of wrapper application. Is it possible to build it?
Less complicated task is too interesting: make this wrapper using .NET (both host and client are written in .NET language).
Safest way would be to load that plugin into a separate AppDomain which you configure with the security evidence for the requirements you have.
When you create an AppDomain, you can specify exactly the kinds of things code can do in this sandbox. Code that runs there is restricted to the limits you set. But this process can be confusing the first time you do it and may still leave you open to vulnerabilities.
Using AppDomains to isolate assemblies is an interesting process. You'd think you load your plugins into the other AppDomain then use them via proxies in your AppDomain, but its the other way around. They need to use your proxies in their AppDomain. If you fail to understand and do this right, you'll end up loading your plugin code within your main AppDomain and executing it there instead of in the restricted domain. There are lots of gotchas that you'll get bit by (subscribing to events has some interesting side effects) if you don't do things correctly.
I'd suggest prototyping, brush up on the AppDomain chapter in CLR Via C#, and read as much as you can on the subject.
Here's a test app I made to investigate cross-appdomain events.
http://cid-f8be9de57b85cc35.skydrive.live.com/self.aspx/Public/appdomainevents.rar