I am working on a game engine + editor in C#. The idea is that the user writes all his game code in a project that the editor will then load, so it can list the user-defined classes, as well as instantiate them in a scene editor.
Messing around with the concept of (re)loading a DLL, I can load the DLL, instantiate and invoke methods on a class defined in an external assembly, rebuild the external assembly, and load it again in the host, without restarting the host. And this work (atleast it looks like it), however it is not freeing up memory from the previously loaded assembly.
Loader class:
public class Loader
{
// This DLL contains an implementation of ILoadable - ILoadable is declared in a shared project.
private const string AsmPath = #"D:\Dev\AsmLoading\AsmLoadingImpl\bin\Debug\AsmLoadingImpl.dll";
public ILoadable GetExternalLoadable()
{
var asmBytes = File.ReadAllBytes(AsmPath);
Assembly asm = Assembly.Load(asmBytes, null);
var loadableInterface = typeof(ILoadable);
var loadableImpl = asm.GetTypes().First(loadableInterface.IsAssignableFrom);
return (ILoadable)Activator.CreateInstance(loadableImpl);
}
}
Running the GetExternalLoadable() 2 or 3 times, the Task Manager reveals a ~1mb increase in RAM usage in my host program, repeating the same action will increase it further, without it ever decreasing.
Is there any way to work around this? I know Unity is doing similar things, except they actually compile the external assembly themselves, but the Unity editor does not consume additional memory when I trigger the recompilation a few times.
So, what I am trying to accomplish is "live reloading" of external assemblies.
.NET 4 added support for collectible assemblies, that can be unloaded by the GC.
However, these are heavily restricted:
They are intended for dynamic code generation, you cannot just load a .dll file
They cannot define COM interop types, P/Invoke methods, thread-static fields, ...
If you have a .dll that follows these restrictions, you might be able to load and re-emit the IL code so that it looks like a dynamically generated assembly to .NET.
Apart from dynamic methods and collectible assemblies, it is impossible to unload code without unloading the whole AppDomain.
Related
I have an executable that depends on a library. I also have a "loader" application, which loads the executable into a seperate AppDomain (my "sandbox") and runs it there. The library needs initialization from outside the sandbox, so I need to load the library before I load the executable. This works as expected, but when I now load the executable, the library is loaded another time, and the executable uses the uninitialized copy.
This only occurs if I load the library like this:
Assembly.Load(File.ReadAllBytes(assemblyFile.FullName));
instead of this:
Assembly.LoadFrom(assemblyFile.FullName);
However, using LoadFrom locks the file. I need to be able to delete/write the file, because my application needs to be able to reload the entire sandbox and all assemblies in it.
I also tried registering AppDomain.AssemblyResolve, but it is only called if it does not find the file, which isn't exactly what I want ...
Example log output:
Loading library X <- manual load
Initializing library X
Loading executable Y
Loading library X <- bad, i want to use the loaded X!
So my question is: How do I force .net to use the already loaded assembly instead of loading a duplicate?
From the remarks on Assembly.Load(Byte[]):
Note that this method overload always creates a new Assembly object with its own mapping.
You can't use that overload if you want to reuse your loaded assemblies (without extra work anyway - the Framework won't automatically do it for you here).
You might be able to use the Assembly.Load(string) or just Type.GetType(string) methods here, but I suspect that's still going to end up locking files you want to modify. I'm not really sure how to make sense of modifying those files at run time to be honest though - what's the expected behavior if you delete or change a loaded assembly? Reload it? Have the modified code entered into memory?
You might need to create some kind of assembly caching mechanism of your own. If the assemblies aren't that large and you can afford to keep them in memory, it might be something as simple as a Dictionary<string, Assembly> - and just check if the dictionary has your assembly before loading it, otherwise load it using Assembly.Load(Byte[]) the way you are now.
I ended up modifying my AppDomainSetup of the sandbox:
domainSetup.DisallowApplicationBaseProbing = true;
Now, AssemblyResolve will be called everytime (no autodiscover for assemblies). Now I can just load assemblies from a byte[] and cache them (thanks to #DanField who suggested caching assemblies)
I have a C# application which interfaces with some hardware (USB device) as follows:
C# application -> intermediate DLL -> hardware DLL -> hardware. The intermediate DLL and hardware DLL are supplied with the USB device so I have no control over these.
The intermediate DLL is the only which which I need to include in the VS project as this is what I call. The hardware DLL is then in the same directory so must be found automatically.
A new version of hardware device is now released with a different hardware DLL. The old DLL is not compatible with the new hardware and the new DLL is not compatible with the old hardware.
How can I make my application work with both pieces of hardware? I guess that I need to load and unload each DLL as required?
Here's what I do for a similar problem. I have a chunk of code that I want to work with, but I have to load the dll at runtime. So I refer to it in my project, but I don't put it in the same directory as the rest of my assemblies. Instead, in the consuming code, I have some code that looks like this:
// constructor called from a static constructor elsewhere
MyDllLoader(string hardwareFolder) {
_hardwareFolder = hardwareFolder;
AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(CurrentDomain_AssemblyResolve);
SeeIfAlreadyLoaded();
}
private void SeeIfAlreadyLoaded() {
// if the assembly is still in the current app domain then the AssemblyResolve event will
// never fire.
// Since we need to know where the assembly is, we have to look for it
// here.
Assembly[] assems = AppDomain.CurrentDomain.GetAssemblies();
foreach (Assembly am in assems)
{
// if it matches, just mark the local _loaded as true and get as much
// other information as you need
}
}
System.Reflection.Assembly CurrentDomain_AssemblyResolve(object sender, ResolveEventArgs args) {
string name = args.Name;
if (name.StartsWith("Intermediate.dll,"))
{
string candidatePath = Path.Combine(_hardwareFolder, "Intermediate.dll");
try {
Assembly assem = Assembly.LoadFrom(candidatePath);
if (assem != null) {
_location = candidateFolder;
_fullPath = candidatePath;
_loaded = true;
return assem;
}
}
catch (Exception err) {
sb.Append(err.Message);
}
}
return null;
}
There's another solution too - it's complicated, but I've done it and done the work for you. You declare an abstract class, say MyHardwareAbstraction, that has the signatures of the methods that you want and you code against that interface. Then you write some code which given a path to an assembly, loads it and dynamically defines a new class that matches MyHardwareAbstraction and makes it map onto an instance of the actual object that you want. I wrote a blog several years ago on how to do this.
The nice thing about doing this is that you use the abstract type in your code and work against that and then the adapter compiler will, at run time, compile a new class that will complete that abstract type using some other type as the target type. It's fairly efficient too.
I you want both dll to coexist in the program, you'll have to use AppDomains, as explained here.
Else, you can simply use LoadLibrary after the user has made a clear choice about what version he needs ?
Edit:
If the intermediate DLL is a .Net Assembly, you can use the method mentioned here to specify where to look for your intermediate DLL before you call any method that uses the intermediate DLL, without having to change your existing code.
then you must not directly reference the DLL in your C# project, because .Net Assemblies are discovered and loaded before your Main method is even called. Instead, you must dynamically load the intermediate DLL using AppDomain or other methods, then use the library via reflection, or by using dynamic objects.
Apparently, this would make programming very cumbersome. However, there is an alternative method. You can write a launcher program, that loads your original application (you can load .exe files as libraries), and invokes the Main method of your original program reflectively. To make sure that the correct intermediate DLL is loaded, you can use the method mentioned here, while your launcher program is loading your original application.
The following discussion still applies to the hardware DLL.
The following is valid if:
You need only one version of the dll at a time (during the entire period your application runs), and
The two versions of the intermediate DLLs have exactly the same API.
According to MSDN, the DLL Search Path includes directories specified under the PATH environment variable. (http://msdn.microsoft.com/en-us/library/7d83bc18%28v=vs.80%29.aspx). Hence, you may put the two versions of the intermediate DLLs under seperate sub-directories under your application directory, but with exactly the same name under each directory, for example:
bin\
hardware-intermediate-v1\
intermediate.dll
hardware-intermediate-v2\
intermediate.dll
Then, at start up, after your application has determined which version to use, you may add one of the above directories to your PATH environment variable,
using System;
using System.Reflection;
using System.IO;
...
Environment.SetEnvironmentVariable(
"PATH",
Environment.GetEnvironmentVariable("PATH") + ";" +
Path.GetDirectoryName(Assembly.GetEntryAssembly().Location) +
"\\hardware-intermediate-v1"
);
Then, calls to P-Invoke methods (DLLImport) will result in the corresponding version of the DLL to be loaded. To immediately load all DLLs, you may refer to DllImport, how to check if the DLL is loaded?.
However, if you wish to use the two version of the DLLs together without restarting your application, or if there are any API difference between the two DLLs on the level of method name and/or parameter count/type, you must create two seperate sets of P-Invoke methods, each binding to its corresponding version of the intermediate DLL.
I am trying to make a C++/MFC application (so called "Update Application") that can extract AssemblyVersion information from other applications written in C#/.NET 3.5.
So I can update my C# program according to the version information I can get.
I am pretty sure this is possible, but I don't know which way would be the best way.
I would like to know some techniques or keywords that I can search on the web.
A direct explanation would be appreciated, too.
Here's roughly how we do something similar in a native C++ app.
Compile with /clr. You can do this project-wide or just on selected C++ files, but as far as I remember there were complications doing it selectively and so we just did it project-wide. Also #include <vcclr.h> wherever appropriate.
You'll need to learn about app domains. The main thing here is that once you've loaded an assembly into a domain, you can't unload the assembly except by unloading the entire domain. Since you want to load an assembly, query its version, and then let it go, you'll probably want to create a temporary domain and load into this.
In our system we have a managed C++ class called ModelLoader to load object models, query their version info, and discard them - just like what you want to do. This class is the pivotal element in our managed/unmanaged marshaling.
The code in the ModelLoader has to execute in the temporary domain, because we want it to load the target assemblies there and then unload the domain However, the main app is already running in the main domain and so it needs to be able to marshal method calls across to the ModelLoader in the temp domain. So ModelLoader inherits System::MarshalByRefObject, which allows the .NET runtime to do all the marshaling magic.
So the basic steps are something like this:
Load the assembly that contains the code for ModelLoader. In our system this is built into our main unmanaged .EXE and so we just use Reflection::Assembly::GetExecutingAssembly() to get a handle to it. If your equivalent of ModelLoader is in a separate assembly then you'll have to load it somehow. But since you probably won't need to unload this assembly you can load it into the main domain.
Create a temporary domain.
Create an instance of your ModelLoader class (obviously it will have a different name in your system) within the temporary domain.
Marshal a handle to that new instance back to your main domain.
Use the marshaled handle from within your main domain to execute code in the temp domain.
Unload the temporary domain.
So, in code:
AppDomain ^domain = AppDomain::CreateDomain(L"temp domain");
Assembly ^assembly = Assembly::GetExecutingAssembly();
ObjectHandle ^handle = domain->CreateInstanceFrom(
assembly->Location,L"ModeLoader");
Object ^o = handle->Unwrap();
ModelLoader ^loader = dynamic_cast<ModelLoader^>(o);
loader->GetAssemblyVersion(...);
AppDomain::Unload(domain);
To save you some head-scratching, the namespaces involved are:
System::AppDomain
System::Reflection::Assembly
System::Runtime::Remoting::ObjectHandle
System::Object
Within your ModelLoader, you'll want to load the target assembly and query its version info. Compared to all the other stuff, this is straightforward:
void ModelLoader::GetAssemblyVersion(const wchar_t *filename, AssemblyName ^name)
{
Assembly ^assembly = Assembly::Load(gcnew String(filename));
name = assembly->GetName();
}
(I made this function up just now, so it might not be quite right.)
Another thing to watch out for is assembly resolution. This is how the assembly loader resolves assembly names to DLL files. This is quite a large field in its own right, so I won't talk any more about it right now. (And in any case I'm no expert.) To get started, just make sure that all the assemblies you want to load are in your main app directory and I think you'll be more or less OK. Then when you have the basic loading working, you can worry about more sophisticated resolution.
I have an IronPython 2.6/2.7 script I am writing which imports a lot of assemblies.
In other words, at the top of the script it does this...
clr.AddReference( "System.Xml" )
import System.Xml
Except it doesn't do this for 1 assembly, but for 10 assemblies.
Some of the modules are built-in .NET assembllies and some are assemblies I have made.
I'd like to simplify my script so that it loads one assembly that I will build. I want to then call a method in that assembly that will do the "AddReference" and "import" for the 10 assemblies. The primary goal of all this is to minimize the length/complexity of the script.
So in the end I would see it working like this
clr.AddReferenceToFileAndPath( "d:\\myassembly" )
import MyAssembly
MyAssembly.ImportAllAssembliesIReallyWant()
My core problemis despite reading all the information I could find on ScriptRuntime, ScriptEngine, scopes, etc. - I still can't figure out how to write a method in "MyAssembly" that affects what modules are loaded in the calling script.
One way to go about this would be to create a built-in module which does this. You can do this with:
[assembly: PythonModule("mymodule", typeof(MyModuleType)]
public static class MyModuleType {
[SpecialName]
public static void PerformModuleReload(PythonContext context, PythonDictionary dict) {
context.DomainManager.LoadAssembly(typeof(TypeInAssemblyToLoad));
}
}
Just add appropriate LoadAssembly calls for all of the assemblies you care about. The assembly could also populate members in dict that you want available.
Another (and possibly simpler) way would be to simply have a .py file which does all of the clr.AddReference calls you need and have every module import that one file. The import mechanism will do the appropriate caching so it will only load once but will ensure all of the assemblies are available for each module which needs them.
I think that the only way to do this is going to be to access the ScriptEngine from within your ImportAllAssemblies() method and execute the commands that would normally be executed. You should be able to dynamically generate the statements based on what assemblies you want to load and that are referenced etc.
Hope that helps point you in the right direction.
Let’s say that I have two C# applications - game.exe (XNA, needs to support Xbox 360) and editor.exe (XNA hosted in WinForms) - they both share an engine.dll assembly that does the vast majority of the work.
Now let’s say that I want to add some kind of C#-based scripting (it’s not quite "scripting" but I’ll call it that). Each level gets its own class inherited from a base class (we’ll call it LevelController).
These are the important constraints for these scripts:
They need to be real, compiled C# code
They should require minimal manual "glue" work, if any
They must run in the same AppDomain as everything else
For the game - this is pretty straight forward: All the script classes can be compiled into an assembly (say, levels.dll) and the individual classes can be instanced using reflection as needed.
The editor is much harder. The editor has the ability to "play the game" within the editor window, and then reset everything back to where it started (which is why the editor needs to know about these scripts in the first place).
What I am trying to achieve is basically a "reload script" button in the editor that will recompile and load the script class associated with the level being edited and, when the user presses the "play" button, create an instance of the most recently compiled script.
The upshot of which will be a rapid edit-test workflow within the editor (instead of the alternative - which is to save the level, close the editor, recompile the solution, launch the editor, load the level, test).
Now I think I have worked out a potential way to achieve this - which itself leads to a number of questions (given below):
Compile the collection of .cs files required for a given level (or, if need be, the whole levels.dll project) into a temporary, unique-named assembly. That assembly will need to reference engine.dll. How to invoke the compiler this way at runtime? How to get it to output such an assembly (and can I do it in memory)?
Load the new assembly. Will it matter that I am loading classes with the same name into the same process? (I am under the impression that the names are qualified by assembly name?)
Now, as I mentioned, I can’t use AppDomains. But, on the other hand, I don’t mind leaking old versions of script classes, so the ability to unload isn’t important. Unless it is? I’m assuming that loading maybe a few hundred assemblies is feasible.
When playing the level, instance the class that is inherited from LevelController from the specific assembly that was just loaded. How to do this?
And finally:
Is this a sensible approach? Could it be done a better way?
UPDATE: These days I use a far simpler approach to solve the underlying problem.
There is now a rather elegant solution, made possible by (a) a new feature in .NET 4.0, and (b) Roslyn.
Collectible Assemblies
In .NET 4.0, you can specify AssemblyBuilderAccess.RunAndCollect when defining a dynamic assembly, which makes the dynamic assembly garbage collectible:
AssemblyBuilder ab = AppDomain.CurrentDomain.DefineDynamicAssembly(
new AssemblyName("Foo"), AssemblyBuilderAccess.RunAndCollect);
With vanilla .NET 4.0, I think that you need to populate the dynamic assembly by writing methods in raw IL.
Roslyn
Enter Roslyn: Roslyn lets you compile raw C# code into a dynamic assembly. Here's an example, inspired by these two blog posts, updated to work with the latest Roslyn binaries:
using System;
using System.Reflection;
using System.Reflection.Emit;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
namespace ConsoleApplication1
{
public static class Program
{
private static Type CreateType()
{
SyntaxTree tree = SyntaxTree.ParseText(
#"using System;
namespace Foo
{
public class Bar
{
public static void Test()
{
Console.WriteLine(""Hello World!"");
}
}
}");
var compilation = Compilation.Create("Hello")
.WithOptions(new CompilationOptions(OutputKind.DynamicallyLinkedLibrary))
.AddReferences(MetadataReference.CreateAssemblyReference("mscorlib"))
.AddSyntaxTrees(tree);
ModuleBuilder helloModuleBuilder = AppDomain.CurrentDomain
.DefineDynamicAssembly(new AssemblyName("FooAssembly"), AssemblyBuilderAccess.RunAndCollect)
.DefineDynamicModule("FooModule");
var result = compilation.Emit(helloModuleBuilder);
return helloModuleBuilder.GetType("Foo.Bar");
}
static void Main(string[] args)
{
Type fooType = CreateType();
MethodInfo testMethod = fooType.GetMethod("Test");
testMethod.Invoke(null, null);
WeakReference weak = new WeakReference(fooType);
fooType = null;
testMethod = null;
Console.WriteLine("type = " + weak.Target);
GC.Collect();
Console.WriteLine("type = " + weak.Target);
Console.ReadKey();
}
}
}
In summary: with collectible assemblies and Roslyn, you can compile C# code into an assembly that can be loaded into an AppDomain, and then garbage collected (subject to a number of rules).
Check out the namespaces around Microsoft.CSharp.CSharpCodeProvider and System.CodeDom.Compiler.
Compile the collection of .cs files
Should be pretty straightforward like http://support.microsoft.com/kb/304655
Will it matter that I am loading classes with the same name into the same process?
Not at all. It's just names.
instance the class that is inherited from LevelController.
Load the assembly that you created something like Assembly.Load etc. Query the type you want to instanciate using reflection. Get the constructor and call it.
Well, you want to be able to edit things on the fly, right? that's your goal here isn't it?
When you compile assemblies and load them there's now way to unload them unless you unload your AppDomain.
You can load pre-compiled assemblies with the Assembly.Load method and then invoke the entry point through reflection.
I would consider the dynamic assembly approach. Where you through your current AppDomain say that you want to create a dynamic assembly. This is how the DLR (dynamic language runtime) works. With dynamic assemblies you can create types that implement some visible interface and call them through that. The back side of working with dynamic assemblies is that you have to provide correct IL yourself, you can't simply generate that with the built in .NET compiler, however, I bet the Mono project has a C# compiler implementation you might wanna check out. They already have a C# interpreter which reads in a C# source file and compiles that and executes it, and that's definitely handled through the System.Reflection.Emit API.
I'm not sure about the garbage collection here though, because when it comes to dynamic types I think the runtime doesn't release them because they can be referenced at any time. Only if the dynamic assembly itself is destroyed and no references exist to that assembly would it be reasonable to free that memory. If you're and re-generating a lot of code make sure that the memory is, at some point, collected by the GC.
If the language was Java the answer would be to use JRebel. Since it isn't, the answer is to raise enough noise to show there's demand for this. It might require some sort of alternate CLR or 'c# engine project template' and VS IDE working in coordination etc.
I doubt there's a lot of scenarios where this is "must have" but there's many in which it would save lot of time as you could get away with less infrastructure and quicker turnaround for things that aren't going to be used for long. (Yeah there's some who argue to over-engineer things because they'll be used 20+ years but the problem with that is when you need to do some massive change, it'll likely be as expensive as rebuilding the whole thing from scratch. So it comes down to whether to spend money now or later. Since it's not know for sure the project will become business critical later and might require large changes later anyhow, the argument to me is to use "KISS"-principle and have the complexities of live-editing in the IDE,CLR/runtime and so forth instead of building it into each app where it might be useful later. Certainly some defensive programming and practises will be needed to modify some live service using this sort of feature. As Erlang devs are said to be doing)