Why does clr.AddReference fail on second IronPython engine? - c#

When I try to add a reference to an IronPython engine instance, the reference get's added to the references as expected. If I create another instance of the engine, the AddReference executes without an error, but the reference is not added to the references and import statements fail with "no module named ...".
var engine = Python.CreateEngine();
dynamic clr = engine.Runtime.GetClrModule();
clr.AddReference("IronPython.StdLib");
var references = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // ok
var source = "import pydoc\npydoc.plain(pydoc.render_doc(str))";
var result = engine.Execute<string>(source);
Debug.Assert(result.StartsWith("Python Library Documentation")); // ok
var engine2 = Python.CreateEngine();
dynamic clr2 = engine2.Runtime.GetClrModule();
clr2.AddReference("IronPython.StdLib");
var references2 = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references2.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // fails
result = engine.Execute<string>(source); // throws ImportException "no module named pydoc"
Debug.Assert(result.StartsWith("Python Library Documentation"));
I tried with the binary release of IronPython 2.7.5 (installed to GAC) and C# 4.5, IronPython.StdLib is a precompiled DLL of the Python standard lib with pyc.
I also tried with self compiled IronPython 2.7.5 and 2.7.6 from github, but there the first engine.execute already fails with "no module named pydoc" although the reference gets added.
Am I doing something wrong or is it just a bug?

A colleague found out the reason for the failure. I post it here, in case someone else stumbles across this issue.
You need to load the assembly after adding the reference:
var engine = Python.CreateEngine();
dynamic clr = engine.Runtime.GetClrModule();
clr.AddReference("IronPython.StdLib");
// load assembly into engine
var assembly = Assembly.LoadFrom("IronPython.StdLib.dll");
engine.Runtime.LoadAssembly(assembly);
var references = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // ok
var source = "import pydoc\npydoc.plain(pydoc.render_doc(str))";
var result = engine.Execute<string>(source);
Debug.Assert(result.StartsWith("Python Library Documentation")); // ok
var engine2 = Python.CreateEngine();
dynamic clr2 = engine2.Runtime.GetClrModule();
clr2.AddReference("IronPython.StdLib");
// load assembly into engine2
engine2.Runtime.LoadAssembly(assembly);
var references2 = (IEnumerable<Assembly>)clr.References;
Debug.Assert(references2.Any(asm => asm.FullName.StartsWith("IronPython.StdLib"))); // does not fail
result = engine.Execute<string>(source); // does not throw any more
Debug.Assert(result.StartsWith("Python Library Documentation"));

Related

ASP.NET MVC - Dynamically load views from external DLL

I have a use case in an application I'm writing where I have logic in an external DLL that is loaded dynamically. Now I need to add the ability to display shared views inside the ASP.NET MVC view that resides in the external DLL.
What I've done so far is to add the following in my ConfigureServices method:
UriBuilder uri = new UriBuilder(Assembly.GetEntryAssembly().CodeBase);
string fullPath = Uri.UnescapeDataString(uri.Path);
var mainDirectory = Path.GetDirectoryName(fullPath);
var assemblyFilePath = Path.Combine(mainDirectory, "MyLogic.dll");
var asmStream = File.OpenRead(assemblyFilePath);
var assembly = AssemblyLoadContext.Default.LoadFromStream(asmStream);
var part = new AssemblyPart(assembly);
services.AddControllersWithViews().ConfigureApplicationPartManager(apm => apm.ApplicationParts.Add(part));
This works fine as long as the DLL is added as a reference to the project. If I remove the reference then I'm getting an error in my application when I try to load the partial view:
InvalidOperationException: The partial view 'MyView' was not found. The following locations were searched: /Views/Consent/MyView.cshtml /Views/Shared/MyView.cshtml
What I tried doing is to list all known view of the application using the following code:
var feature = new ViewsFeature();
applicationPartManager.PopulateFeature(feature);
var views = feature.ViewDescriptors.Select(x => x.RelativePath).ToList();
What I see is that when I add the DLL as a reference in the project I see MyView.cshtml in the list, and if not then I don't see it - and the above error makes sense.
But my use case dictates that the loaded DLL is not referenced. Is there a way to add the views from it when it's not a reference?
Mystery solved... instead of ConfigureApplicationPartManager need to use AddApplicationPart like this:
UriBuilder uri = new UriBuilder(Assembly.GetEntryAssembly().CodeBase);
string fullPath = Uri.UnescapeDataString(uri.Path);
var mainDirectory = Path.GetDirectoryName(fullPath);
var assemblyFilePath = Path.Combine(mainDirectory, "Risco.Auth.Demo.Bridge.dll");
var asmStream = File.OpenRead(assemblyFilePath);
var assembly = AssemblyLoadContext.Default.LoadFromStream(asmStream);
services.AddControllersWithViews().AddApplicationPart(assembly);

Why do I get Microsoft.Scripting.SyntaxErrorException: unexpected token ',' when importing libraries using IronPython

I'm using IronPython v 2.7.8.1 in VS 2017. I've installed Python27, 36 and 37. I've tried switching between the various environments in VS. I've tried adding the search paths to the libraries of these installs. The python code will work if I run it in the interpreter. Trying to run the same code in VS throws: "Microsoft.Scripting.SyntaxErrorException: unexpected token ',' ". If I test a python script that doesn't include imports it will work? Is there a specific way python has to be installed to work IronPython? This is the C# Code:
class CallPython
{
public void PatchParameter(string parameter)
{
var FilePath = (#"C:\Users\Pac\Downloads\MvcAuth\MvcAuth\TwurlPy\TwurlPy\SendDirectMsg.py");
var engine = Python.CreateEngine(); // Extract Python language engine from their grasp
ICollection<string> searchPaths = engine.GetSearchPaths();
searchPaths.Add(#"C:\Users\Pac\AppData\Local\Programs\Python\Python37\Lib");
searchPaths.Add(#"C:\Users\Pac\AppData\Local\Programs\Python\Python37\Lib\site-packages");
engine.SetSearchPaths(searchPaths);
var scope = engine.CreateScope(); // Introduce Python namespace
(scope)
var d = new Dictionary<string, object>
{
{ "text", text},
{ "userID", userID},
};
// Add some sample parameters. Notice that there is no need in
// specifically setting the object type, interpreter will do that part for us
// in the script properly with high probability
scope.SetVariable("params", d); // This will be the name of the
// dictionary in python script, initialized with previously created .NET
// Dictionary
ScriptSource source = engine.CreateScriptSourceFromFile(FilePath);
// Load the script
object result = source.Execute(scope);
parameter = scope.GetVariable<string>("parameter"); // To get the
// finally set variable 'parameter' from the python script
return;
}
}
This is the Python script. If I comment out the import statements it works using IronPython, but of course I need them...
import twitter
import requests
import sys
parameter = "test"
def SendDM(text, userID, access_token, access_secret):
consumer_key = 'YkopsCQjEXccccccccccccccccccZvA9yy'
consumer_secret = 'TQVCoccccccccccccccccccct7y8VfmE'
access_token_key = access_token
access_token_secret = access_secret
api = twitter.Api(
consumer_key=consumer_key,
consumer_secret=consumer_secret,
access_token_key=access_token_key,
access_token_secret=access_token_secret)
send_msg = api.PostDirectMessage(text, user_id=userID)
print (send_msg)
return
SendDM(text, userID, access_token, access_secret)

Memory leak when compiled script frequenlty changing

I'm working on an module of MVVM-application based on the MVVMLight.Messenger and Roslyn scripts. Idea is to give to user ability to modify business logic, attach and detach user scripts from objects. The problem is that when you frequently change the executable code of scripts, the size of the memory occupied by the application grows.
I use code from this answer.
var initial = CSharpCompilation.Create("Existing")
.AddReferences(MetadataReference.CreateFromFile(typeof(object).Assembly.Location))
.AddSyntaxTrees(SyntaxFactory.ParseSyntaxTree(CHANGED_METHOD_BODY));
var method = initial.GetSymbolsWithName(x => x == "Script").Single();
// 1. get source
var methodRef = method.DeclaringSyntaxReferences.Single();
var methodSource = methodRef.SyntaxTree.GetText().GetSubText(methodRef.Span).ToString();
// 2. compile in-memory as script
var compilation = CSharpCompilation.CreateScriptCompilation("Temp")
.AddReferences(initial.References)
.AddSyntaxTrees(SyntaxFactory.ParseSyntaxTree(methodSource, CSharpParseOptions.Default.WithKind(SourceCodeKind.Script)));
using (var dll = new MemoryStream())
using (var pdb = new MemoryStream())
{
compilation.Emit(dll, pdb);
// 3. load compiled assembly
assembly = Assembly.Load(dll.ToArray(), pdb.ToArray());
var methodBase = assembly.GetType("Script").GetMethod(method.Name, new Type[0]);
// 4. get il or even execute
MethodBody il = methodBase.GetMethodBody();
return methodBase;
}
It happens because I execute Assembly.Load every time I compile a changed script. The old assembly remains apparently in the CLR.
Are there any approaches to implement changing logic and prevent memory leaks?

How to correctly load a WF4 workflow from XAML?

Short version:
How do I load a WF4 workflow from XAML?
Important detail: The code that loads the workflow shouldn't need to know beforehand which types are used in the workflow.
Long version:
I am having a very hard time loading a WF4 workflow from the XAML file create by Visual Studio.
My scenario is that I want to put this file into the database to be able to modify it centrally without recompiling the Workflow invoker.
I am currently using this code:
var xamlSchemaContext = new XamlSchemaContext(GetWFAssemblies());
var xmlReaderSettings = new XamlXmlReaderSettings();
xmlReaderSettings.LocalAssembly = typeof(WaitForAnySoundActivity).Assembly;
var xamlReader = ActivityXamlServices.CreateBuilderReader(
new XamlXmlReader(stream, xmlReaderSettings),
xamlSchemaContext);
var activityBuilder = (ActivityBuilder)XamlServices.Load(xamlReader);
var activity = activityBuilder.Implementation;
var validationResult = ActivityValidationServices.Validate(activity);
This gives me a whole lot of errors, which fall into two categories:
Category 1:
Types from my assemblies are not known, although I provided the correct assemblies to the constructor of XamlSchemaContext.
ValidationError { Message = Compiler error(s) encountered processing expression "GreetingActivationResult.WrongPin".
'GreetingActivationResult' is not declared. It may be inaccessible due to its protection level.
, Source = 10: VisualBasicValue, PropertyName = , IsWarning = False }
This can be solved by using the technique described here, which basically adds the assemblies and namespaces of all used types to some VisualBasicSettings instance:
var settings = new VisualBasicSettings();
settings.ImportReferences.Add(new VisualBasicImportReference
{
Assembly = typeof(GreetingActivationResult).Assembly.GetName().Name,
Import = typeof(GreetingActivationResult).Namespace
});
// ...
VisualBasic.SetSettings(activity, settings);
// ... Validate here
This works but makes the whole "dynamic loading" part of the Workflow a joke, as the code still needs to know all used namespaces.
Question 1: Is there another way to get rid of these validation errors without the need to know beforehand which namespaces and assemblies are used?
Category 2:
All my input arguments are unknown. I can see them just fine in activityBuilder.Properties but I still get validation errors saying they are unknown:
ValidationError { Message = Compiler error(s) encountered processing expression
"Pin".
'Pin' is not declared. It may be inaccessible due to its protection level.
, Source = 61: VisualBasicValue, PropertyName = , IsWarning = False }
No solution so far.
Question 2: How to tell WF4 to use the arguments defined in the XAML file?
Question 2:
You can´t execute an ActivityBuilder, it´s just for design. You have to load a DynamicActivity (only through ActivityXamlServices). It should work that way (without using a special XamlSchemaContext), but you must have loaded all used assemblies in advance (placing them in the bin directory should also work, so far about Question 1, DynamicActivity might make things a little bit easier):
var dynamicActivity = ActivityXamlServices.Load(stream) as DynamicActivity;
WorkflowInvoker.Invoke(dynamicActivity);
In general, I got the impression that you´re trying to implement your own "ActivityDesigner" (like VS). I tried this myself, and it was quite hard to deal with DynamicActivity and ActivityBuilder (as DynamicActivity is not serializable but ActivityBuilder cannot be executed), so I ended up with an own activity type that internally converts one type into the other. If you want to have a look at my results, read the last sections of this article.
I have a project that does this - the assemblies are also stored in a database.
When it is time to instantiate a workflow instance I do the following:
Download the assemblies from the database to a cache location
Create a new AppDomain passing the assembly paths into it.
From the new AppDomain load each assembly - you may also need to load assemblies required by your hosting environment too.
I didn't need to mess around with VisualBasic settings - at least as far as I can see having taken a quick look in my code but I'm sure I've seen it somewhere...
In my case while I don't know the input names or types, the caller is expected to have built a request that contains the input names and values (as strings) which are then converted into the correct types via a reflection helper class.
At this point I can instantiate the workflow.
My AppDomain initialisation code looks like this:
/// <summary>
/// Initializes a new instance of the <see cref="OperationWorkflowManagerDomain"/> class.
/// </summary>
/// <param name="requestHandlerId">The request handler id.</param>
public OperationWorkflowManagerDomain(Guid requestHandlerId)
{
// Cache the id and download dependent assemblies
RequestHandlerId = requestHandlerId;
DownloadAssemblies();
if (!IsIsolated)
{
Domain = AppDomain.CurrentDomain;
_manager = new OperationWorkflowManager(requestHandlerId);
}
else
{
// Build list of assemblies that must be loaded into the appdomain
List<string> assembliesToLoad = new List<string>(ReferenceAssemblyPaths);
assembliesToLoad.Add(Assembly.GetExecutingAssembly().Location);
// Create new application domain
// NOTE: We do not extend the configuration system
// each app-domain reuses the app.config for the service
// instance - for now...
string appDomainName = string.Format(
"Aero Operations Workflow Handler {0} AppDomain",
requestHandlerId);
AppDomainSetup ads =
new AppDomainSetup
{
AppDomainInitializer = new AppDomainInitializer(DomainInit),
AppDomainInitializerArguments = assembliesToLoad.ToArray(),
ApplicationBase = AppDomain.CurrentDomain.BaseDirectory,
PrivateBinPathProbe = null,
PrivateBinPath = PrivateBinPath,
ApplicationName = "Aero Operations Engine",
ConfigurationFile = Path.Combine(
AppDomain.CurrentDomain.BaseDirectory, "ZenAeroOps.exe.config")
};
// TODO: Setup evidence correctly...
Evidence evidence = AppDomain.CurrentDomain.Evidence;
Domain = AppDomain.CreateDomain(appDomainName, evidence, ads);
// Create app-domain variant of operation workflow manager
// TODO: Handle lifetime leasing correctly
_managerProxy = (OperationWorkflowManagerProxy)Domain.CreateInstanceAndUnwrap(
Assembly.GetExecutingAssembly().GetName().Name,
typeof(OperationWorkflowManagerProxy).FullName);
_proxyLease = (ILease)_managerProxy.GetLifetimeService();
if (_proxyLease != null)
{
//_proxyLease.Register(this);
}
}
}
The download assemblies code is easy enough:
private void DownloadAssemblies()
{
List<string> refAssemblyPathList = new List<string>();
using (ZenAeroOpsEntities context = new ZenAeroOpsEntities())
{
DbRequestHandler dbHandler = context
.DbRequestHandlers
.Include("ReferenceAssemblies")
.FirstOrDefault((item) => item.RequestHandlerId == RequestHandlerId);
if (dbHandler == null)
{
throw new ArgumentException(string.Format(
"Request handler {0} not found.", RequestHandlerId), "requestWorkflowId");
}
// If there are no referenced assemblies then we can host
// in the main app-domain
if (dbHandler.ReferenceAssemblies.Count == 0)
{
IsIsolated = false;
ReferenceAssemblyPaths = new string[0];
return;
}
// Create folder
if (!Directory.Exists(PrivateBinPath))
{
Directory.CreateDirectory(PrivateBinPath);
}
// Download assemblies as required
foreach (DbRequestHandlerReferenceAssembly dbAssembly in dbHandler.ReferenceAssemblies)
{
AssemblyName an = new AssemblyName(dbAssembly.AssemblyName);
// Determine the local assembly path
string assemblyPathName = Path.Combine(
PrivateBinPath,
string.Format("{0}.dll", an.Name));
// TODO: If the file exists then check it's SHA1 hash
if (!File.Exists(assemblyPathName))
{
// TODO: Setup security descriptor
using (FileStream stream = new FileStream(
assemblyPathName, FileMode.Create, FileAccess.Write))
{
stream.Write(dbAssembly.AssemblyPayload, 0, dbAssembly.AssemblyPayload.Length);
}
}
refAssemblyPathList.Add(assemblyPathName);
}
}
ReferenceAssemblyPaths = refAssemblyPathList.ToArray();
IsIsolated = true;
}
And finally the AppDomain initialisation code:
private static void DomainInit(string[] args)
{
foreach (string arg in args)
{
// Treat each string as an assembly to load
AssemblyName an = AssemblyName.GetAssemblyName(arg);
AppDomain.CurrentDomain.Load(an);
}
}
Your proxy class needs to implement MarshalByRefObject and serves as your communication link between your app and the new appdomain.
I find that I am able to load workflows and get the root activity instance without any problem.
EDIT 29/07/12 **
Even if you only store the XAML in the database you will need to track the referenced assemblies. Either your list of referenced assemblies will tracked in an additional table by name or you will have to upload (and obviously support download) the assemblies referenced by the workflow.
Then you may simply enumerate all the reference assemblies and add ALL namespaces from ALL public types to the VisualBasicSettings object - like this...
VisualBasicSettings vbs =
VisualBasic.GetSettings(root) ?? new VisualBasicSettings();
var namespaces = (from type in assembly.GetTypes()
select type.Namespace).Distinct();
var fullName = assembly.FullName;
foreach (var name in namespaces)
{
var import = new VisualBasicImportReference()
{
Assembly = fullName,
Import = name
};
vbs.ImportReferences.Add(import);
}
VisualBasic.SetSettings(root, vbs);
Finally don't forget to add namespaces from the environment assemblies - I add namespaces from the following assemblies:
mscorlib
System
System.Activities
System.Core
System.Xml
So in summary:
1. Track the assembly referenced by the user's workflow (since you will be rehosting the workflow designer this will be trivial)
2. Build a list of assemblies from which namespaces will be imported - this will be a union of the default environment assemblies and the user referenced assemblies.
3. Update the VisualBasicSettings with the namespaces and reapply to the root activity.
You will need to do this in the project that executes workflow instances and in the project that rehosts the workflow designer.
One system that I know which does the same job that you are trying to do is the Team Foundation 2010's build system. When you execute a custom build workflow on a controller, you need to point the build controller to a path in TFS where you keep your custom assemblies. The controller then recursively loads up all the assemblies from that location as it starts processing the workflow.
You mentioned that you need to keep the file in a database. Can you not also store the location or meta data information about the required assemblies in the same database and use Reflection to load them recursively before you invoke your workflow?
You can then selectively add/remove assemblies from this path without having to alter the code that dynamically load assemblies using the
var settings = new VisualBasicSettings();
settings.ImportReferences.Add(new VisualBasicImportReference
{
Assembly = typeof(GreetingActivationResult).Assembly.GetName().Name,
Import = typeof(GreetingActivationResult).Namespace
});
// ...
VisualBasic.SetSettings(activity, settings);
// ... Validate here
approach.
This is the way how I load xaml embeded resource (default workflow) to a Workflow Designer:
//UCM.WFDesigner is my assembly name,
//Resources.Flows is the folder name,
//and DefaultFlow.xaml is the xaml name.
private const string ConstDefaultFlowFullName = #"UCM.WFDesigner.Resources.Flows.DefaultFlow.xaml";
private void CreateNewWorkflow(object param)
{
//loading default activity embeded resource
using (var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(ConstDefaultFlowFullName))
{
StreamReader sReader = new StreamReader(stream);
string content = sReader.ReadToEnd();
//createion ActivityBuilder from string
ActivityBuilder activityBuilder = XamlServices.Load( ActivityXamlServices
.CreateBuilderReader(new XamlXmlReader(new StringReader(content)))) as ActivityBuilder;
//loading new ActivityBuilder to Workflow Designer
_workflowDesigner.Load(activityBuilder);
OnPropertyChanged("View");
}
}

How to invoke a function written in a javascript file from C# using IronJS

I just download the Iron JS and after doing some 2/3 simple programs using the Execute method, I am looking into the ExecuteFile method.
I have a Test.js file whose content is as under
function Add(a,b)
{
var result = a+b;
return result;
}
I want to invoke the same from C# using Iron JS. How can I do so? My code so far
var o = new IronJS.Hosting.CSharp.Context();
dynamic loadFile = o.ExecuteFile(#"d:\test.js");
var result = loadFile.Add(10, 20);
But loadfile variable is null (path is correct)..
How to invoke JS function ,please help... Also searching in google yielded no help.
Thanks
The result of the execution is going to be null, because your script does not return anything.
However, you can access the "globals" object after the script has run to grab the function.
var o = new IronJS.Hosting.CSharp.Context();
o.ExecuteFile(#"d:\test.js");
dynamic globals = o.Globals;
var result = globals.Add(10, 20);
EDIT: That particular version will work with the current master branch, and in an up-coming release, but is not quite what we have working with the NuGet package. The slightly more verbose version that works with IronJS version 0.2.0.1 is:
var o = new IronJS.Hosting.CSharp.Context();
o.ExecuteFile(#"d:\test.js");
var add = o.Globals.GetT<FunctionObject>("Add");
var result = add.Call(o.Globals, 10D, 20D).Unbox<double>();

Categories