We Keep Coding

Python and .Net Integration options

I want to integrate Python with C#. I found two approaches using Interprocess communication and IronPython
Interprocess communication requires Python.exe to be installed on all client machines so not a viable solution.
We started using IronPython, but it only supports 2.7 python version for now. We are using 3.7 version.
Following code we tried using IronPython:
private void BtnJsonPy_Click(object sender, EventArgs e)
{
// 1. Create Engine
var engine = Python.CreateEngine();
//2. Provide script and arguments
var script = #"C:\Users\user\source\path\repos\SamplePy\SamplePy2\SamplePy2.py"; // provide full path
var source = engine.CreateScriptSourceFromFile(script);
// dummy parameters to send Python script
int x = 3;
int y = 4;
var argv = new List<string>();
argv.Add("");
argv.Add(x.ToString());
argv.Add(y.ToString());
engine.GetSysModule().SetVariable("argv", argv);
//3. redirect output
var eIO = engine.Runtime.IO;
var errors = new MemoryStream();
eIO.SetErrorOutput(errors, Encoding.Default);
var results = new MemoryStream();
eIO.SetOutput(results, Encoding.Default);
//4. Execute script
var scope = engine.CreateScope();
var lib = new[]
{
"C:\\Users\\user\\source\\repos\\SamplePy\\CallingWinForms\\Lib",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9\\lib",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.2.7.9",
"C:\\Users\\user\\source\\repos\\SamplePy\\packages\\IronPython.StdLib.2.7.9"
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32\\Lib",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\python.exe",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37 - 32",
//"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37-32\\DLLs"
};
engine.SetSearchPaths(lib);
engine.ExecuteFile(script, scope);
//source.Execute(scope);
//5. Display output
string str(byte[] x1) => Encoding.Default.GetString(x1);
Console.WriteLine("Errrors");
Console.WriteLine(str(errors.ToArray()));
Console.WriteLine();
Console.WriteLine("Results");
Console.WriteLine(str(results.ToArray()));
lblAns.Text = str(results.ToArray());
}
The problem sometimes is, to do heavy Machine Learning programming we need to add "Modules". These Modules would be dependent on other modules. This increases point 4, Execute Scripts part of code, as more data path of that module has to be given here var lib = new[] and also some modules are not supported with Iron Python as well (for e.g. modules concerning OCR operations etc.)
Due to these limitations I found Pythonnet which also helps in integrating .net applications with Python. But I am new to it, so want some ideas on implementing the same, and code samples available, and is it feasible or recommended to use with Python 3.7
I checked that setting up Pythonnet is cumbersome initially, so want help or steps on how to set up the same. Also would like to know if in future Iron Python would support Python 3.X as well or not.

I am not familiar with IronPython, but I use pythonnet quite a lot for the same purpose - integrate Python with C#, so I can elaborate on that.
The advantage of using pythonnet for your purposes is having all the CPython packages available for you to use (numpy, scipy, pandas, Theano, Keras, scikit-learn etc), but avoiding the overhead of calling python.exe as separate process (pythonnet works by loading pythonXY.dll into your process).
Pay attention that pythonnet also requires to have stand-alone Python availiable, but you can use Embeddable Python package which is very light-weight and can be distributed with your application.
pythonnnet supports Python 3.7, but the published NuGet packages are only for Python 3.5. You have several choices to obtain pythonnet for Python 3.7:
Download pythonnet wheel package from PyPi and extract Python.Runtime.dll from it
Download NuGet package from pythonnet appveyor build artifacts, as advised on pythonnet installation wiki
Build from sources
Important note: pythonnet version has to match your Python version and bitness. For example, if you are using Python 3.7 32-bit, download pythonnet-2.4.0-cp37-cp37m-win32.whl. If your Python is 64-bit, download pythonnet-2.4.0-cp37-cp37m-win_amd64.whl. Your C# project platform target also has to match (x86 for 32-bit or x64 for 64-bit).
Code sample with similar functionality to what you have posted, using pythonnet (tested with Python 3.7.4 on Windows 7 and pythonnet NuGet from latest build artifacts):
private void Test()
{
// Setup all paths before initializing Python engine
string pathToPython = #"C:\Users\user\AppData\Local\Programs\Python\Python37-32";
string path = pathToPython + ";" +
Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", pathToPython, EnvironmentVariableTarget.Process);
var lib = new[]
{
#"C:\Users\user\source\path\repos\SamplePy\SamplePy2",
Path.Combine(pathToPython, "Lib"),
Path.Combine(pathToPython, "DLLs")
};
string paths = string.Join(";", lib);
Environment.SetEnvironmentVariable("PYTHONPATH", paths, EnvironmentVariableTarget.Process);
using (Py.GIL()) //Initialize the Python engine and acquire the interpreter lock
{
try
{
// import your script into the process
dynamic sampleModule = Py.Import("SamplePy");
// It is more maintainable to communicate with the script with
// function parameters and return values, than using argv
// and input/output streams.
int x = 3;
int y = 4;
dynamic results = sampleModule.sample_func(x, y);
Console.WriteLine("Results: " + results);
}
catch (PythonException error)
{
// Communicate errors with exceptions from within python script -
// this works very nice with pythonnet.
Console.WriteLine("Error occured: ", error.Message);
}
}
}
SamplePy.py:
def sample_func(x, y):
return x*y

How to load referenced script libraries from a Roslyn script file?

I figured out I cannot load one script library from another easily:
module.csx
string SomeFunction() {
return "something";
}
script.csx
ExecuteFile("module.csx");
SomeFunction() <-- causes compile error "SomeFunction" does not exist
This is because the compiler does not know of module.csx at the time it compiles script.csx afaiu. I can add another script to load the two files from that one, and that will work. However thats not that pretty.
Instead I like to make my scripthost check for a special syntax "load module" within my scripts, and execute those modules before actual script execution.
script.csx
// load "module.csx"
SomeFunction()
Now, with some basic string handling, I can figure out which modules to load (lines that contains // load ...) and load that files (gist here https://gist.github.com/4147064):
foreach(var module in scriptModules) {
session.ExecuteFile(module);
}
return session.Execute(script)
But - since we're talking Roslyn, there should be some nice way to parse the script for the syntax I'm looking for, right?
And it might even exist a way to handle module libraries of code?

Currently in Roslyn there is no way to reference another script file. We are considering moving #load from being a host command of the Interactive Window to being a part of the language (like #r), but it isn't currently implemented.
As to how to deal with the strings, you could parse it normally, and then look for pre-processor directives that are of an unknown type and delve into the structure that way.

Support for #load in script files has been added as of https://github.com/dotnet/roslyn/commit/f1702c.
This functionality will be available in Visual Studio 2015 Update 1.

Include the script:
#load "common.csx"
...
And configure the source resolver when you run the scripts:
Script<object> script = CSharpScript.Create(code, ...);
var options = ScriptOptions.Default.WithSourceResolver(new SourceFileResolver(new string[] { }, baseDirectory));
var func = script.WithOptions(options).CreateDelegate()
...

Embedding lua in my C# application : require "luainterface" fails

I downloaded the newest version of LuaInterface from their site, and referenced LuaInterface.dll and Lua51.dll. The interpreter itself works fine, but when I try to require("luainterface"), I get this exception :
error loading module 'luainterface' from file '.\luainterface.dll':
The specified procedure could not be found.
Here's the example code which produces this behavior :
static void Main(string[] args)
{
Lua lua = new Lua();
lua.DoFile("test.lua");
}
The test.lua script just has this :
luanet = require("luainterface");
Also, I've made sure that LUA_PATH points to where luanet.dll is.
What could be the problem?

For .dll's are looked for in LUA_CPATH in any case...
http://www.lua.org/manual/5.1/manual.html#pdf-package.cpath

Embedding IronPython in a C# application - import error on urllib

I have a Python file with as content:
import re
import urllib
class A(object):
def __init__(self, x):
self.x = x
def getVal(self):
return self.x
def __str__(self):
return "instance of A with value '%s'" % (self.getVal())
I also have a simple C# console project with the following code:
engine = Python.CreateEngine();
ScriptSource source = engine.CreateScriptSourceFromFile("test.py");
ScriptScope scope = engine.CreateScope();
ObjectOperations op = engine.Operations;
source.Execute(scope); // class object created
object klaz = scope.GetVariable("A"); // get the class object
object instance = op.Call(klaz, "blabla waarde"); // create the instance
object method = op.GetMember(instance, "getVal"); // get a method
string result = (string)op.Call(method); // call method and get result (9)
Console.WriteLine("Result: " + result); //output: 'Result: blabla waarde'
(I got this from this stackoverflow querstion and answer)
If I leave out the the import urllib statement in the Python file everything works fine. (meaning it finds the re module)
But as soon as i either add import urllib or import urllib2 I get the following exception:
ImportException was unhandled
No module named urllib
So somehow it can't find the urllib. I checked the IronPython lib folder and both urllib and urllib 2 are definitely there.
The same exception gets thrown when I import urllib in the C# code. (engine.ImportModule("urllib");)
Any ideas?
I'd like to manage the imports in the python code and not in the C# code.
(So I'd like to avoid stuff like this: engine.ImportModule("urllib");)
Edit:
Some extra info on what I'm actually going to use this for (maybe someone has an alternative):
I will have a main C# application and the python scripts will be used as extensions or plugins for the main application.
I'm using Python so that I don't need to compile any of the plugins.

I believe that 'Lib' being on sys.path from the interactive console is actually done inside ipy.exe - and when embedding you will have to add the path manually. Either the engine or the runtime has a 'SetSourcePaths' (or similar) method that will allow you to do this.

I face the same problem. Following "Tom E's" suggestion in the comments to fuzzyman's reply I could successfully resolve the issue. The issue seems to be it is not able resolve the location of the urllib.py. We need to set it.
You can check the following link for the question and answer.

The version of CPython you're importing from must match your IronPython version. Use CPython v2.5 for IronPython 2.0, or v2.6 for IronPython 2.6.
Try this:
import sys
sys.path.append(r'\c:\python26\lib') # adjust to whatever version of CPython you have installed.
import urllib

Call function from DLL with non-static path

I have a DLL that I need to access methods from.
In most cases like this I just use [DllImport] to access methods from unmanaged assemblies, but the problem with that in this situation is that it requires the path to the DLL at instantiation time, so a constant string.
This particular DLL is one that gets installed with my application and I can't guarantee where it will be after the program is installed (I'd rather not put it somewhere static like %SystemRoot%).
So is there a way in C# that I can declare and use a method from a DLL at runtime with a variable path?
Any ideas or suggestions would be greatly appreciated!

This is a bit of hack, but since you say that you can find the path to the dll at runtime, why not copy it to your current working directory before you use any of the functions? That way, the dll will exist next to your exe and will be found by LoadLibrary. No need for any additional path in your DllImport.
The only other way to use a method from a dynamic path is to do this:
1) Do the necessary P/Invoke signatures for LoadLibrary & GetProcAddress
2) Load the library from the desired path (LoadLibrary)
3) Find the desired function (GetProcAddress)
4) Cast the pointer to a delegate Marshal.GetDelegateForFunctionPointer
5) Invoke it.
Of course, you will need to declare a delegate for each function you want to "import" in this way since you have to cast the pointer to a delegate.

Don't use a path at all. Windows uses a default method of searching for DLLs when trying to dynamically or statically load a function from it.
The exact search logic is documented at MSDN in the docs for LoadLibrary - basically, if the DLL is just used by your app, put in the same folder as your application during the install and don't worry about it. If it's a commonly used DLL, put it somewhere in the folder structure searched by LoadLibrary() and it'll get found.

I had a similar situation. I use DLLs from a SDK that is installed on the machine. I get the directory location of the DLLs from that SDKs registry key. I set the DLL location on the executing users PATH variable (only temporary modification). Basically it allows you to set a dynamic path for the DLL you want to invoke, so it don't have to be from registry. Mind that the PATH var is the last place Windows looks for DLLs. But on the other hand, it does not change the other places Windows looks for DLLs.
Example:
API i want to call, on the DLL:
[DllImport("My.DLL")]
private static extern IntPtr ApiCall(int param);
Get the registry key (you need using Microsoft.Win32;):
private static string GetRegistryKeyPath() {
string environmentPath = null;
using (var rk = Registry.LocalMachine.OpenSubKey(#"SOFTWARE\SOMENNAME"))
{
if (rk != null)
{
environmentPath = rk.GetValue("Path(or whatever your key is)").ToString();
}
if (string.IsNullOrEmpty(environmentPath))
{
Log.Warn(
string.Format("Path not found in Windows registry, using key: {0}. Will default to {1}",
#"SOFTWARE\SOMETHING", #"C:\DefaultPath"));
environmentPath = #"C:\DefaultPath";
}
}
return environmentPath;
}
Add the path of the DLL on the PATH var (Concat() is found in Linq):
void UpdatePath(IEnumerable<string> paths){
var path = new[] { Environment.GetEnvironmentVariable("PATH") ?? "" };
path = path.Concat(paths);
string modified = string.Join(Path.PathSeparator.ToString(), path);
Environment.SetEnvironmentVariable("PATH", modified);
}
Start Using the API call:
var sdkPathToAdd = GetRegistryKeyPath();
IList<string> paths = new List<string>
{
Path.Combine(sdkPathToAdd),
Path.Combine("c:\anotherPath")
};
UpdatePath(paths);
//Start using
ApiCall(int numberOfEyes);