We Keep Coding

Memory leak when compiled script frequenlty changing

I'm working on an module of MVVM-application based on the MVVMLight.Messenger and Roslyn scripts. Idea is to give to user ability to modify business logic, attach and detach user scripts from objects. The problem is that when you frequently change the executable code of scripts, the size of the memory occupied by the application grows.
I use code from this answer.
var initial = CSharpCompilation.Create("Existing")
.AddReferences(MetadataReference.CreateFromFile(typeof(object).Assembly.Location))
.AddSyntaxTrees(SyntaxFactory.ParseSyntaxTree(CHANGED_METHOD_BODY));
var method = initial.GetSymbolsWithName(x => x == "Script").Single();
// 1. get source
var methodRef = method.DeclaringSyntaxReferences.Single();
var methodSource = methodRef.SyntaxTree.GetText().GetSubText(methodRef.Span).ToString();
// 2. compile in-memory as script
var compilation = CSharpCompilation.CreateScriptCompilation("Temp")
.AddReferences(initial.References)
.AddSyntaxTrees(SyntaxFactory.ParseSyntaxTree(methodSource, CSharpParseOptions.Default.WithKind(SourceCodeKind.Script)));
using (var dll = new MemoryStream())
using (var pdb = new MemoryStream())
{
compilation.Emit(dll, pdb);
// 3. load compiled assembly
assembly = Assembly.Load(dll.ToArray(), pdb.ToArray());
var methodBase = assembly.GetType("Script").GetMethod(method.Name, new Type[0]);
// 4. get il or even execute
MethodBody il = methodBase.GetMethodBody();
return methodBase;
}
It happens because I execute Assembly.Load every time I compile a changed script. The old assembly remains apparently in the CLR.
Are there any approaches to implement changing logic and prevent memory leaks?

Adding references to current project to CSharpCodeProvider

I am trying to compile a string using CodeDom.
Dictionary<string, string> providerOptions = new Dictionary<string, string>
{
{"CompilerVersion", "v3.5"}
};
CSharpCodeProvider provider = new CSharpCodeProvider(providerOptions);
CompilerParameters compilerParams = new CompilerParameters
{
GenerateInMemory = true,
GenerateExecutable = false
};
compilerParams.ReferencedAssemblies.Add(Assembly.GetEntryAssembly().Location);
My problem is: In my compiled code I need to use classes from the project I am creating the code in. So I tried to add my current assembly as a reference, however it gives me the following error:
Metadata file 'Path\to\my\executable\MyProject.exe' could not be
opened -- 'An attempt was made to load a program with an incorrect
format.'
Anybody knows where my mistake is?

I had the same need and I solved this way:
compilerParams.ReferencedAssemblies.AddRange(Assembly.GetExecutingAssembly().GetReferencedAssemblies().Select(a => a.Name + ".dll").ToArray());
the +".dll" worked with me.
if you need to get the full path to the Assembly file you need to load them into a new AppDomain and then get the .Location of each one of them.

Why does clr.AddReference fail on second IronPython engine?

When I try to add a reference to an IronPython engine instance, the reference get's added to the references as expected. If I create another instance of the engine, the AddReference executes without an error, but the reference is not added to the references and import statements fail with "no module named ...".
var engine = Python.CreateEngine();
dynamic clr = engine.Runtime.GetClrModule();
clr.AddReference("IronPython.StdLib");
var references = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // ok
var source = "import pydoc\npydoc.plain(pydoc.render_doc(str))";
var result = engine.Execute<string>(source);
Debug.Assert(result.StartsWith("Python Library Documentation")); // ok
var engine2 = Python.CreateEngine();
dynamic clr2 = engine2.Runtime.GetClrModule();
clr2.AddReference("IronPython.StdLib");
var references2 = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references2.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // fails
result = engine.Execute<string>(source); // throws ImportException "no module named pydoc"
Debug.Assert(result.StartsWith("Python Library Documentation"));
I tried with the binary release of IronPython 2.7.5 (installed to GAC) and C# 4.5, IronPython.StdLib is a precompiled DLL of the Python standard lib with pyc.
I also tried with self compiled IronPython 2.7.5 and 2.7.6 from github, but there the first engine.execute already fails with "no module named pydoc" although the reference gets added.
Am I doing something wrong or is it just a bug?

A colleague found out the reason for the failure. I post it here, in case someone else stumbles across this issue.
You need to load the assembly after adding the reference:
var engine = Python.CreateEngine();
dynamic clr = engine.Runtime.GetClrModule();
clr.AddReference("IronPython.StdLib");
// load assembly into engine
var assembly = Assembly.LoadFrom("IronPython.StdLib.dll");
engine.Runtime.LoadAssembly(assembly);
var references = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // ok
var source = "import pydoc\npydoc.plain(pydoc.render_doc(str))";
var result = engine.Execute<string>(source);
Debug.Assert(result.StartsWith("Python Library Documentation")); // ok
var engine2 = Python.CreateEngine();
dynamic clr2 = engine2.Runtime.GetClrModule();
clr2.AddReference("IronPython.StdLib");
// load assembly into engine2
engine2.Runtime.LoadAssembly(assembly);
var references2 = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references2.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // does not fail
result = engine.Execute<string>(source); // does not throw any more
Debug.Assert(result.StartsWith("Python Library Documentation"));

How to correctly load a WF4 workflow from XAML?

Short version:
How do I load a WF4 workflow from XAML?
Important detail: The code that loads the workflow shouldn't need to know beforehand which types are used in the workflow.
Long version:
I am having a very hard time loading a WF4 workflow from the XAML file create by Visual Studio.
My scenario is that I want to put this file into the database to be able to modify it centrally without recompiling the Workflow invoker.
I am currently using this code:
var xamlSchemaContext = new XamlSchemaContext(GetWFAssemblies());
var xmlReaderSettings = new XamlXmlReaderSettings();
xmlReaderSettings.LocalAssembly = typeof(WaitForAnySoundActivity).Assembly;
var xamlReader = ActivityXamlServices.CreateBuilderReader(
new XamlXmlReader(stream, xmlReaderSettings),
xamlSchemaContext);
var activityBuilder = (ActivityBuilder)XamlServices.Load(xamlReader);
var activity = activityBuilder.Implementation;
var validationResult = ActivityValidationServices.Validate(activity);
This gives me a whole lot of errors, which fall into two categories:
Category 1:
Types from my assemblies are not known, although I provided the correct assemblies to the constructor of XamlSchemaContext.
ValidationError { Message = Compiler error(s) encountered processing expression "GreetingActivationResult.WrongPin".
'GreetingActivationResult' is not declared. It may be inaccessible due to its protection level.
, Source = 10: VisualBasicValue, PropertyName = , IsWarning = False }
This can be solved by using the technique described here, which basically adds the assemblies and namespaces of all used types to some VisualBasicSettings instance:
var settings = new VisualBasicSettings();
settings.ImportReferences.Add(new VisualBasicImportReference
{
Assembly = typeof(GreetingActivationResult).Assembly.GetName().Name,
Import = typeof(GreetingActivationResult).Namespace
});
// ...
VisualBasic.SetSettings(activity, settings);
// ... Validate here
This works but makes the whole "dynamic loading" part of the Workflow a joke, as the code still needs to know all used namespaces.
Question 1: Is there another way to get rid of these validation errors without the need to know beforehand which namespaces and assemblies are used?
Category 2:
All my input arguments are unknown. I can see them just fine in activityBuilder.Properties but I still get validation errors saying they are unknown:
ValidationError { Message = Compiler error(s) encountered processing expression
"Pin".
'Pin' is not declared. It may be inaccessible due to its protection level.
, Source = 61: VisualBasicValue, PropertyName = , IsWarning = False }
No solution so far.
Question 2: How to tell WF4 to use the arguments defined in the XAML file?

Question 2:
You can´t execute an ActivityBuilder, it´s just for design. You have to load a DynamicActivity (only through ActivityXamlServices). It should work that way (without using a special XamlSchemaContext), but you must have loaded all used assemblies in advance (placing them in the bin directory should also work, so far about Question 1, DynamicActivity might make things a little bit easier):
var dynamicActivity = ActivityXamlServices.Load(stream) as DynamicActivity;
WorkflowInvoker.Invoke(dynamicActivity);
In general, I got the impression that you´re trying to implement your own "ActivityDesigner" (like VS). I tried this myself, and it was quite hard to deal with DynamicActivity and ActivityBuilder (as DynamicActivity is not serializable but ActivityBuilder cannot be executed), so I ended up with an own activity type that internally converts one type into the other. If you want to have a look at my results, read the last sections of this article.

I have a project that does this - the assemblies are also stored in a database.
When it is time to instantiate a workflow instance I do the following:
Download the assemblies from the database to a cache location
Create a new AppDomain passing the assembly paths into it.
From the new AppDomain load each assembly - you may also need to load assemblies required by your hosting environment too.
I didn't need to mess around with VisualBasic settings - at least as far as I can see having taken a quick look in my code but I'm sure I've seen it somewhere...
In my case while I don't know the input names or types, the caller is expected to have built a request that contains the input names and values (as strings) which are then converted into the correct types via a reflection helper class.
At this point I can instantiate the workflow.
My AppDomain initialisation code looks like this:
/// <summary>
/// Initializes a new instance of the <see cref="OperationWorkflowManagerDomain"/> class.
/// </summary>
/// <param name="requestHandlerId">The request handler id.</param>
public OperationWorkflowManagerDomain(Guid requestHandlerId)
{
// Cache the id and download dependent assemblies
RequestHandlerId = requestHandlerId;
DownloadAssemblies();
if (!IsIsolated)
{
Domain = AppDomain.CurrentDomain;
_manager = new OperationWorkflowManager(requestHandlerId);
}
else
{
// Build list of assemblies that must be loaded into the appdomain
List<string> assembliesToLoad = new List<string>(ReferenceAssemblyPaths);
assembliesToLoad.Add(Assembly.GetExecutingAssembly().Location);
// Create new application domain
// NOTE: We do not extend the configuration system
// each app-domain reuses the app.config for the service
// instance - for now...
string appDomainName = string.Format(
"Aero Operations Workflow Handler {0} AppDomain",
requestHandlerId);
AppDomainSetup ads =
new AppDomainSetup
{
AppDomainInitializer = new AppDomainInitializer(DomainInit),
AppDomainInitializerArguments = assembliesToLoad.ToArray(),
ApplicationBase = AppDomain.CurrentDomain.BaseDirectory,
PrivateBinPathProbe = null,
PrivateBinPath = PrivateBinPath,
ApplicationName = "Aero Operations Engine",
ConfigurationFile = Path.Combine(
AppDomain.CurrentDomain.BaseDirectory, "ZenAeroOps.exe.config")
};
// TODO: Setup evidence correctly...
Evidence evidence = AppDomain.CurrentDomain.Evidence;
Domain = AppDomain.CreateDomain(appDomainName, evidence, ads);
// Create app-domain variant of operation workflow manager
// TODO: Handle lifetime leasing correctly
_managerProxy = (OperationWorkflowManagerProxy)Domain.CreateInstanceAndUnwrap(
Assembly.GetExecutingAssembly().GetName().Name,
typeof(OperationWorkflowManagerProxy).FullName);
_proxyLease = (ILease)_managerProxy.GetLifetimeService();
if (_proxyLease != null)
{
//_proxyLease.Register(this);
}
}
}
The download assemblies code is easy enough:
private void DownloadAssemblies()
{
List<string> refAssemblyPathList = new List<string>();
using (ZenAeroOpsEntities context = new ZenAeroOpsEntities())
{
DbRequestHandler dbHandler = context
.DbRequestHandlers
.Include("ReferenceAssemblies")
.FirstOrDefault((item) => item.RequestHandlerId == RequestHandlerId);
if (dbHandler == null)
{
throw new ArgumentException(string.Format(
"Request handler {0} not found.", RequestHandlerId), "requestWorkflowId");
}
// If there are no referenced assemblies then we can host
// in the main app-domain
if (dbHandler.ReferenceAssemblies.Count == 0)
{
IsIsolated = false;
ReferenceAssemblyPaths = new string[0];
return;
}
// Create folder
if (!Directory.Exists(PrivateBinPath))
{
Directory.CreateDirectory(PrivateBinPath);
}
// Download assemblies as required
foreach (DbRequestHandlerReferenceAssembly dbAssembly in dbHandler.ReferenceAssemblies)
{
AssemblyName an = new AssemblyName(dbAssembly.AssemblyName);
// Determine the local assembly path
string assemblyPathName = Path.Combine(
PrivateBinPath,
string.Format("{0}.dll", an.Name));
// TODO: If the file exists then check it's SHA1 hash
if (!File.Exists(assemblyPathName))
{
// TODO: Setup security descriptor
using (FileStream stream = new FileStream(
assemblyPathName, FileMode.Create, FileAccess.Write))
{
stream.Write(dbAssembly.AssemblyPayload, 0, dbAssembly.AssemblyPayload.Length);
}
}
refAssemblyPathList.Add(assemblyPathName);
}
}
ReferenceAssemblyPaths = refAssemblyPathList.ToArray();
IsIsolated = true;
}
And finally the AppDomain initialisation code:
private static void DomainInit(string[] args)
{
foreach (string arg in args)
{
// Treat each string as an assembly to load
AssemblyName an = AssemblyName.GetAssemblyName(arg);
AppDomain.CurrentDomain.Load(an);
}
}
Your proxy class needs to implement MarshalByRefObject and serves as your communication link between your app and the new appdomain.
I find that I am able to load workflows and get the root activity instance without any problem.
EDIT 29/07/12 **
Even if you only store the XAML in the database you will need to track the referenced assemblies. Either your list of referenced assemblies will tracked in an additional table by name or you will have to upload (and obviously support download) the assemblies referenced by the workflow.
Then you may simply enumerate all the reference assemblies and add ALL namespaces from ALL public types to the VisualBasicSettings object - like this...
VisualBasicSettings vbs =
VisualBasic.GetSettings(root) ?? new VisualBasicSettings();
var namespaces = (from type in assembly.GetTypes()
select type.Namespace).Distinct();
var fullName = assembly.FullName;
foreach (var name in namespaces)
{
var import = new VisualBasicImportReference()
{
Assembly = fullName,
Import = name
};
vbs.ImportReferences.Add(import);
}
VisualBasic.SetSettings(root, vbs);
Finally don't forget to add namespaces from the environment assemblies - I add namespaces from the following assemblies:
mscorlib
System
System.Activities
System.Core
System.Xml
So in summary:
1. Track the assembly referenced by the user's workflow (since you will be rehosting the workflow designer this will be trivial)
2. Build a list of assemblies from which namespaces will be imported - this will be a union of the default environment assemblies and the user referenced assemblies.
3. Update the VisualBasicSettings with the namespaces and reapply to the root activity.
You will need to do this in the project that executes workflow instances and in the project that rehosts the workflow designer.

One system that I know which does the same job that you are trying to do is the Team Foundation 2010's build system. When you execute a custom build workflow on a controller, you need to point the build controller to a path in TFS where you keep your custom assemblies. The controller then recursively loads up all the assemblies from that location as it starts processing the workflow.
You mentioned that you need to keep the file in a database. Can you not also store the location or meta data information about the required assemblies in the same database and use Reflection to load them recursively before you invoke your workflow?
You can then selectively add/remove assemblies from this path without having to alter the code that dynamically load assemblies using the
var settings = new VisualBasicSettings();
settings.ImportReferences.Add(new VisualBasicImportReference
{
Assembly = typeof(GreetingActivationResult).Assembly.GetName().Name,
Import = typeof(GreetingActivationResult).Namespace
});
// ...
VisualBasic.SetSettings(activity, settings);
// ... Validate here
approach.

This is the way how I load xaml embeded resource (default workflow) to a Workflow Designer:
//UCM.WFDesigner is my assembly name,
//Resources.Flows is the folder name,
//and DefaultFlow.xaml is the xaml name.
private const string ConstDefaultFlowFullName = #"UCM.WFDesigner.Resources.Flows.DefaultFlow.xaml";
private void CreateNewWorkflow(object param)
{
//loading default activity embeded resource
using (var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(ConstDefaultFlowFullName))
{
StreamReader sReader = new StreamReader(stream);
string content = sReader.ReadToEnd();
//createion ActivityBuilder from string
ActivityBuilder activityBuilder = XamlServices.Load( ActivityXamlServices
.CreateBuilderReader(new XamlXmlReader(new StringReader(content)))) as ActivityBuilder;
//loading new ActivityBuilder to Workflow Designer
_workflowDesigner.Load(activityBuilder);
OnPropertyChanged("View");
}
}

Is there a way to build a new type during Runtime?

I am going to ask a question that might sound weird.
Is there a way to build a new class during Runtime? Or at least, add a new property to an existing class.
I mean creating a class that doesn't exist and not an instance of an existing class. I could later on use reflections to load and use this class.

Adding a property to an existing type is not possible, but you can create a new type at runtime using Reflection.Emit. It's pretty complicated stuff, and it goes something like this:
AssemblyBuilder assemblyBuilder = Thread.GetDomain().DefineDynamicAssembly(
assemblyName , AssemblyBuilderAccess.Run, assemblyAttributes);
ModuleBuilder moduleBuilder = assemblyBuilder.DefineDynamicModule("ModuleName");
TypeBuilder typeBuilder = moduleBuilder.DefineType(
"MyNamespace.TypeName" , TypeAttributes.Public);
typeBuilder.DefineDefaultConstructor(MethodAttributes.Public);
// Add a method
newMethod = typeBuilder.DefineMethod("MethodName" , MethodAttributes.Public);
ILGenerator ilGen = newMethod.GetILGenerator();
// Create IL code for the method
ilGen.Emit(...);
// ...
// Create the type itself
Type newType = typeBuilder.CreateType();
This code is just a sample. It could contain errors.
You can also generate classes by compiling C# source code at runtime using System.CodeDom, but I don't know a lot about that.

Take a look at the System.Reflection.Emit namespace. I've never used it myself but the classes in this namespace can be used to generate IL (intermediate language).

This is not a weird question - in some cases it might be very useful. For instance I use this technique for performance tests sometimes:
public static Type[] DynamicTypes;
public void CreateObjects()
{
var codeNamespace = new CodeNamespace( "DynamicClasses" );
codeNamespace.Imports.Add( new CodeNamespaceImport( "System" ) );
codeNamespace.Imports.Add( new CodeNamespaceImport( "System.ComponentModel" ) );
for( var i = 0; i < 2000; i++ )
{
var classToCreate = new CodeTypeDeclaration( "DynamicClass_" + i )
{
TypeAttributes = TypeAttributes.Public
};
var codeConstructor1 = new CodeConstructor
{
Attributes = MemberAttributes.Public
};
classToCreate.Members.Add( codeConstructor1 );
codeNamespace.Types.Add( classToCreate );
}
var codeCompileUnit = new CodeCompileUnit();
codeCompileUnit.Namespaces.Add( codeNamespace );
var compilerParameters = new CompilerParameters
{
GenerateInMemory = true,
IncludeDebugInformation = true,
TreatWarningsAsErrors = true,
WarningLevel = 4
};
compilerParameters.ReferencedAssemblies.Add( "System.dll" );
var compilerResults = new CSharpCodeProvider().CompileAssemblyFromDom( compilerParameters, codeCompileUnit );
if( compilerResults == null )
{
throw new InvalidOperationException( "ClassCompiler did not return results." );
}
if( compilerResults.Errors.HasErrors )
{
var errors = string.Empty;
foreach( CompilerError compilerError in compilerResults.Errors )
{
errors += compilerError.ErrorText + "\n";
}
Debug.Fail( errors );
throw new InvalidOperationException( "Errors while compiling the dynamic classes:\n" + errors );
}
var dynamicAssembly = compilerResults.CompiledAssembly;
DynamicTypes = dynamicAssembly.GetExportedTypes();
}

You might take a look at the System.CodeDom namespace. According to one of the pages linked from there:
The .NET Framework includes a mechanism called the Code Document Object Model (CodeDOM) that enables developers of programs that emit source code to generate source code in multiple programming languages at run time, based on a single model that represents the code to render.
I'm not at all an expert in this, I just remembered seeing it on my .NET Framework poster on my wall. :)
Edit: Since writing this answer, I have played with System.CodeDom a bit. I've written a blog post that uses some basic CodeDom that may help those wanting to get started with it.