We Keep Coding

Codegeneration at runtime from a string to console exe is not working in C# .NET6

I have some code which must be able to generated a console application at runtime (Codegeneration with System.CodeDom). I did this already a lot, but in NET 6 now I am struggling with that and the new API. In the code below I try to compile simply from a string. See below the static class with method Start() which then should generates the application.
The compilations seems fine, no errors at the end. But when starting the generated AppCodegenerated.exe, it shows some reference exception with System.Runtime.
Please help, any Idea? Already researched a lot but could not find any useful solution..
//-
I used the Visual Studio 2022 / NET 6 and theses Nuget's:
using Basic.Reference.Assemblies;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Text;
using System.Text;
namespace CompilerSimplified
{
public static class Compiler
{
public static bool Start()
{
string FileName = "AppCodegenerated";
string ExePath = AppDomain.CurrentDomain.BaseDirectory + #"\" + FileName + ".exe";
string code = #"using System; Console.WriteLine(""Hello.""); Console.ReadLine(); ";
// ------- References -------------
// .net platform references
List<MetadataReference> References = new List<MetadataReference>();
foreach (var item in ReferenceAssemblies.Net60) // ReferenceAssemblies from Nuget: Basic.Reference.Assemblies;
References.Add(item);
// or tried this: loop manually through system platform
//string[] fileEntries = Directory.GetFiles(#"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.0\ref\net6.0\", "*.dll");
//foreach (string fileName in fileEntries)
// references.Add(MetadataReference.CreateFromFile(fileName));MetadataReference.CreateFromFile(fileName));
// ------- References END -------------
// delete existing file
if (File.Exists(ExePath))
File.Delete(ExePath);
// compiler options
CSharpCompilationOptions DefaultCompilationOptions =
new CSharpCompilationOptions(outputKind: OutputKind.ConsoleApplication, platform: Platform.AnyCpu)
.WithOverflowChecks(true).WithOptimizationLevel(OptimizationLevel.Release);
// encode soucre code
string sourceCode = SourceText.From(code, Encoding.UTF8).ToString();
// CSharp options
var parsedSyntaxTree = Parse(sourceCode, "", CSharpParseOptions.Default.WithLanguageVersion(LanguageVersion.CSharp10));
// compilation
var compilation = CSharpCompilation.Create(FileName, new SyntaxTree[] { parsedSyntaxTree }, references: References, DefaultCompilationOptions);
var result = compilation.Emit(ExePath);
// return
if (result.Success)
return true;
else
return false;
}
private static SyntaxTree Parse(string text, string filename = "", CSharpParseOptions options = null)
{
var stringText = SourceText.From(text, Encoding.UTF8);
return SyntaxFactory.ParseSyntaxTree(stringText, options, filename);
}
}
}
Above code runs fine without error and exports the AppCodegenerated.exe into the project /bin folder.
Execution of this generated AppCodegenerated.exe shows following on the output console:
Unhandled exception: System.IO.FileNotFoundException:
The file or assembly "System.Runtime, Version = 6.0.0.0, Culture = neutral,
PublicKeyToken = b03f5f7f11d50a3a" or a dependency on it was not found.
The system can not find the stated file.

It is not possible to codegenerate directly a console application like the initial approach above. One possible solution is to generate first a dll (what I mentioned above in the example code is working fine), and from there include that .dll into a .exe, from where the functionality can run.

How to use a dll that returns array pointer in python?

I'm trying to get out an array in a pythonic format [numpy or list] form a DLLwritten in c#/c++. This DLL gives out the pointer to the memory and not the evaluated array. How can I access the memory in a FAST way (I wrote the DLL to speedup the code)?
I have used the pythonnet module and i managed to get the memory pointer as a but I'm failing in extracting the memory pointer as an integer or binary and then to get the data of the output vector.
Here the python code:
import numpy as np
import clr
clr.AddReference(R'...\PyDLL.dll')
from PyDLL import StringLibrary
my_instance = StringLibrary()
x = my_instance.arrayShift(np.array([1,2,3,4,5,6,7,8,9,0]).reshape(-1))
And then the C# code
namespace PyDLL
{
public class StringLibrary
{
public static double[] arrayShift(double[] array)
{
int b = array.Length;
double[] newArray = new double[b];
for(int i=0; i<b; i++)
{
newArray[i] = array[b-i-1];
}
return newArray;
}
}
}
I expect that the output x in python is a list or a np array, while now it is a System.Double[] object.
the expected output of x is [0,9,8,7,6,5,4,3,2,1]
Thank you for any help

After many tries and researches, I managed to solve the problem in this way:
Cpp part:
Open Visual Studio, then create a new project (Dynamic-Link Library).
Name the project and add a new file in the project (Header file) and name it with the same project name.
Insert the following code in the header:
// headerName.h - Contains declarations of math functions
#pragma once
#ifdef HEADERNAME_EXPORTS
#define HEADERNAME_API __declspec(dllexport)
#else
#define HEADERNAME_API __declspec(dllimport)
#endif
// Define the functions that will be called using the DLL
extern "C" HEADERNAME_API void funName1(...);
extern "C" HEADERNAME_API void funName2(...);
extern "C" HEADERNAME_API void funName3(...);
...;
In the folder 'Source Files' add a new file with extension .cpp (call it with the same name as the project), then write the function in this .cpp file. Here an example code:
#include "pch.h" // use stdafx.h in Visual Studio 2017 and earlier COMPULSORY TO ADD
#include "HEADERNAME.h" // COMPULSORY TO ADD
#include lib#1
#include lib#2
#include lib#3
...
using namespace std;
void funName1(...)
{
...
}
void funName2(...)
{
...
}
void funName3(...)
{
...
}
To be able to pass the arrays between python and c++, in my solution, the use of pointers is needed and no return must be used. The output variable is passed as an input variable, initialized in python, using a pointer. In the DLL the memory cells of the output variables are re-wrote and, doing so, when the DLL completes the function, the output is computed and already usable by python without return.
Here an example of how to write a function using pointers for input and output arrays:
void funName(double* inputArray_pointer, double* outputArray_pointer, int extraVariable)
{
double inner_var = 100;
for (int i = 0; i < extraVariable; i++)
{
inner_var = min(inner_var, *(inputArray_pointer+i));
*(outputArray_pointer + i) = inner_var;
}
}
Then rebuild the DLL.
Python part
To use correctly a DLL in the python code, the modules 'ctypes' and 'numpy' must be imported.
Here follows an example of the python code (This will use the example DLL of point (9)):
import numpy as np
import ctypes
cpp_fun = ctypes.CDLL(R'pathToDll/DLLname.dll')
# Define the types of the arguments of the DLL (pointer, int, double, ecc)
cpp_fun.funName.argtypes = [np.ctypeslib.ndpointer(),
np.ctypeslib.ndpointer(),
ctypes.c_int]
# Allocate or define the input variables using numpy or the standard scalar of python (int, float, ecc..)
inputArray = np.array([0,1,2,3,4,5,6,7,8,9,...,1000]).reshape(-1).astype('f8')
outputArray = np.zeros([inputArray.shape[0]])astype('f8')
extraVariable = inputArray.shape[0]
cpp_fun.funName(inputArray, outputArray, extraVariable)
Doing so, in the variable 'outputArray' we will find the result of the calculations performed inside the DLL as a np array structure.

Calling R with RDotNet from C# DLL in Tradelink

I am using R.NET dll found here:
http://rdotnet.codeplex.com/
I am trying to run the sample code from within a custom C# DLL specified for Tradelink (http://code.google.com/p/tradelink)
Here is the code.
using System;
using System.IO;
using System.Linq;
using RDotNet;
using TradeLink.Common;
using TradeLink.API;
using Microsoft.Win32;
namespace Responses
{
class RTLTest : ResponseTemplate
{
////static void Main(string[] args)
public override void GotTick(TradeLink.API.Tick k)
{
//Set the folder in which R.dll locates.
REngine.SetDllDirectory(#"C:\Program Files\R\R-2.12.0\bin\i386");
REngine.SetDllDirectory(GetRPath());
REngine.SetDllDirectory(#"C:\Program Files\R\R-2.14.1\bin\i386");
using (REngine engine = REngine.CreateInstance("RDotNet", new[] { "-q" })) // quiet mode
{
// .NET Framework array to R vector.
NumericVector group1 = engine.CreateNumericVector(new double[] { 30.02, 29.99, 30.11, 29.97, 30.01, 29.99 });
engine.SetSymbol("group1", group1);
// Direct parsing from R script.
NumericVector group2 = engine.EagerEvaluate("group2 <- c(29.89, 29.93, 29.72, 29.98, 30.02, 29.98)").AsNumeric();
// Test difference of mean and get the P-value.
GenericVector testResult = engine.EagerEvaluate("t.test(group1, group2)").AsList();
double p = testResult["p.value"].AsNumeric().First();
string s1 = String.Format("Group1: [{0}]", string.Join(", ", group1));
string s2 = String.Format("Group2: [{0}]", string.Join(", ", group2));
string s3 = String.Format("P-value = {0:0.000}", p);
senddebug(s1);
senddebug(s2);
senddebug(s3);
}
}
}
But I get an error of:
response threw exception: Could not load file or assembly 'R.NET, Version=1.4.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.
Can anyone help out to get around this?

if your ASP.NET application is 64 bit, then this might be useful http://blogs.msdn.com/b/rakkimk/archive/2007/11/03/iis7-running-32-bit-and-64-bit-asp-net-versions-at-the-same-time-on-different-worker-processes.aspx.
Most R packages dlls (except base package) are 32 bit, so it may be a source of the problem.

In C#, how do you reference types from one in-memory assembly inside another?

The example program below compiles two in-memory assemblies. The first compilation works fine. The second one fails because it needs access to a class from the first assembly and the type isn't available.
Specifically: The ReferencedAssemblies member of the CompilerParameters class is a string collection and it is used to load the manifests of the assemblies to obtain their types. It appears the C# compiler gets types strictly from the manifest rather than by using reflection (possibly for performance reasons.) In any case, when an assembly is constructed in memory there is no file and no manifest so the second assembly build fails with an error like this:
COMPILER ERROR: Metadata file 'ax5lw0tl, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' could not be found
Adding an AssemblyResolver event handler doesn't work. I tried this and it looks like it isn't ever called. From what I can tell (and I'm a novice with .Net so bear with me) the compiler only cares about the manifest; it's not actually trying to load the assembly at this time, so AssemblyResolver isn't in the picture.
I could, if desperate, construct my assemblies on disk which would solve the immediate problem to have a physical dll and manifest to read. I would much rather not do this as it leads to having to manage what will become a very large collection of temporary assemblies on disk.
I'm optimistic .Net can do this and, being a novice, I'm simply missing it.
(I hope the spacing comes out ok on the code sample. It seems to render properly in the preview window for a few moments but once the syntax highlighter is done it rerenders and the spacing is incorrect although it remains readable.)
using System;
using System.CodeDom.Compiler;
using System.Reflection;
using System.Collections.Generic;
using Microsoft.CSharp;
namespace AsmCompileTest
{
class Program
{
static Assembly Compile( string code, Assembly referencedAssembly )
{
CompilerParameters cp = new CompilerParameters();
cp.GenerateExecutable = false;
cp.GenerateInMemory = true;
if( null != referencedAssembly )
{
cp.ReferencedAssemblies.Add( referencedAssembly.FullName );
}
CodeDomProvider provider = new CSharpCodeProvider( new Dictionary<string,string> { { "CompilerVersion", "v3.5" } } );
CompilerResults compilerResults = provider.CompileAssemblyFromSource( cp, code );
if( compilerResults.Errors.HasErrors )
{
foreach( CompilerError error in compilerResults.Errors )
{
Console.WriteLine( "COMPILER ERROR: " + error.ErrorText );
}
}
return compilerResults.CompiledAssembly;
}
static string Code1 = "using System;" +
"public class HelloClass" +
" {" +
" public HelloClass() { Console.WriteLine( \"Hello, World!\" ); }" +
" }";
static string Code2 = "using System;" +
"public class TestClass" +
" {" +
" public TestClass() { new HelloClass(); }" +
" }";
static void Main()
{
Assembly asm1 = Compile( Code1, null );
Console.WriteLine( "Compiled: " + asm1.FullName );
asm1.GetType( "HelloClass" ).InvokeMember( String.Empty, BindingFlags.CreateInstance, null, null, null );
Assembly asm2 = Compile( Code2, asm1 );
Console.WriteLine( "Compiled: " + asm2.FullName );
asm2.GetType( "TestClass" ).InvokeMember( String.Empty, BindingFlags.CreateInstance, null, null, null );
}
}
}

Based on documentation found on MSDN and on the code in reflector that I looked at (for the compiler classes) it is not possible to do what you want. The reason is that underneath, the code compiler classes that you are using shell out to the actual compiler.
Also, the code compiler classes are actually generating the temporary files underneath, and based on the code I looked at in reflector, they are not cleaning up the files. So based on that, I would say just generate the file on the disk in a temporary location, and then add reference to it.

Define interfaces in a normal assembly and have classes in each generated assembly implement those interfaces. The generated assemblies will need a reference to the one containing the interfaces, not each other.

Is it possible to serialize a C# code block?

I'm using C# with .NET 3.5. Is it possible to serialize a block of code, transmit it somewhere, deserialize it, and then execute it?
An example usage of this would be:
Action<object> pauxPublish = delegate(object o)
{
if (!(o is string))
{
return;
}
Console.WriteLine(o.ToString());
};
Transmitter.Send(pauxPublish);
With some remote program doing:
var action = Transmitter.Recieve();
action("hello world");
My end goal is to be able to execute arbitrary code in a different process (which has no prior knowledge of the code).

YES!!!
We have done this for a very real case of performance. Doing this at runtime or using a DSL was not an option due to performance.
We compile the code into an assembly, and rip the IL out of the method. We then get all the metadata associated with this method and serialize the whole mess via XML, compress it, and put it in our database.
At re-hydration time, we re-constitute the IL with the metadata using the DynamicMethod class, and execute it.
We do this because of speed. We have thousands of little blocks of code. Unfortunately, to compile a block of code and run it on the fly takes at least 250 ms, which is way too slow for us. We took this approach, and it is working REALLY well. At run-time, it takes an unmeasurable amount of time to reconstitute the method and run it.
Only thing to keep an eye on... Signed assemblies and Unsigned assemblies cannot mix the serialized method data.

You could try to use IronPython in your project. It's trivial to do what you are asking in Python. The Python code could call your C# methods. As for security, you could execute the code in a restricted environment of some kind (one example is RestrictedPython).

Generally speaking that sounds like a really bad idea and a big security hole.
You don't want another process to execute any code. Understand what you really need another process to do and build a little DSL around it.

You could also send it as a string then use the CodeDomProvider to compile it, same result. I have an example bit of code thus:
using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using Microsoft.CSharp;
namespace DynamicCodeApplication
{
class azCodeCompiler
{
private List<string> assemblies;
public azCodeCompiler()
{
assemblies = new List<string>();
scanAndCacheAssemblies();
}
public Assembly BuildAssembly(string code)
{
CodeDomProvider prov = CodeDomProvider.CreateProvider("CSharp");
string[] references = new string[] { }; // Intentionally empty, using csc.rsp
CompilerParameters cp = new CompilerParameters(references)
{
GenerateExecutable = false,
GenerateInMemory = true
};
string path = System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory();
cp.CompilerOptions = "#" + path + #"\csc.rsp";
CompilerResults cr = prov.CompileAssemblyFromSource(cp, code);
foreach (CompilerError err in cr.Errors)
{
Console.WriteLine(err.ToString());
}
return cr.CompiledAssembly;
}
public object ExecuteCode(string code,
string namespacename, string classname,
string functionname, bool isstatic, params object[] args)
{
object returnval = null;
Assembly asm = BuildAssembly(code);
object instance = null;
Type type = null;
if (isstatic)
{
type = asm.GetType(namespacename + "." + classname);
}
else
{
instance = asm.CreateInstance(namespacename + "." + classname);
type = instance.GetType();
}
MethodInfo method = type.GetMethod(functionname);
returnval = method.Invoke(instance, args);
return returnval;
}
private void scanAndCacheAssemblies()
{
/*
foreach (string str in Directory.GetFiles(#"C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727"))
{
if (str.Contains(".dll"))
{
foreach (string st in str.Split(new char[] { '\\' }))
{
if (st.Contains(".dll"))
{
assemblies.Add(st);
}
}
}
}
* */
assemblies.Add("Accessibility.dll");
assemblies.Add("AspNetMMCExt.dll");
assemblies.Add("cscompmgd.dll");
assemblies.Add("CustomMarshalers.dll");
assemblies.Add("IEExecRemote.dll");
assemblies.Add("IEHost.dll");
assemblies.Add("IIEHost.dll");
assemblies.Add("Microsoft.Build.Conversion.dll");
assemblies.Add("Microsoft.Build.Engine.dll");
assemblies.Add("Microsoft.Build.Framework.dll");
assemblies.Add("Microsoft.Build.Tasks.dll");
assemblies.Add("Microsoft.Build.Utilities.dll");
assemblies.Add("Microsoft.Build.VisualJSharp.dll");
assemblies.Add("Microsoft.CompactFramework.Build.Tasks.dll");
assemblies.Add("Microsoft.JScript.dll");
assemblies.Add("Microsoft.VisualBasic.Compatibility.Data.dll");
assemblies.Add("Microsoft.VisualBasic.Compatibility.dll");
assemblies.Add("Microsoft.VisualBasic.dll");
assemblies.Add("Microsoft.VisualBasic.Vsa.dll");
assemblies.Add("Microsoft.Vsa.dll");
assemblies.Add("Microsoft.Vsa.Vb.CodeDOMProcessor.dll");
assemblies.Add("Microsoft_VsaVb.dll");
assemblies.Add("mscorlib.dll");
assemblies.Add("sysglobl.dll");
assemblies.Add("System.configuration.dll");
assemblies.Add("System.Configuration.Install.dll");
assemblies.Add("System.Data.dll");
assemblies.Add("System.Data.OracleClient.dll");
assemblies.Add("System.Data.SqlXml.dll");
assemblies.Add("System.Deployment.dll");
assemblies.Add("System.Design.dll");
assemblies.Add("System.DirectoryServices.dll");
assemblies.Add("System.DirectoryServices.Protocols.dll");
assemblies.Add("System.dll");
assemblies.Add("System.Drawing.Design.dll");
assemblies.Add("System.Drawing.dll");
assemblies.Add("System.EnterpriseServices.dll");
assemblies.Add("System.Management.dll");
assemblies.Add("System.Messaging.dll");
assemblies.Add("System.Runtime.Remoting.dll");
assemblies.Add("System.Runtime.Serialization.Formatters.Soap.dll");
assemblies.Add("System.Security.dll");
assemblies.Add("System.ServiceProcess.dll");
assemblies.Add("System.Transactions.dll");
assemblies.Add("System.Web.dll");
assemblies.Add("System.Web.Mobile.dll");
assemblies.Add("System.Web.RegularExpressions.dll");
assemblies.Add("System.Web.Services.dll");
assemblies.Add("System.Windows.Forms.dll");
assemblies.Add("System.XML.dll");
assemblies.Add("vjscor.dll");
assemblies.Add("vjsjbc.dll");
assemblies.Add("vjslib.dll");
assemblies.Add("vjslibcw.dll");
assemblies.Add("vjssupuilib.dll");
assemblies.Add("vjsvwaux.dll");
assemblies.Add("vjswfc.dll");
assemblies.Add("VJSWfcBrowserStubLib.dll");
assemblies.Add("vjswfccw.dll");
assemblies.Add("vjswfchtml.dll");
assemblies.Add("Accessibility.dll");
assemblies.Add("AspNetMMCExt.dll");
assemblies.Add("cscompmgd.dll");
assemblies.Add("CustomMarshalers.dll");
assemblies.Add("IEExecRemote.dll");
assemblies.Add("IEHost.dll");
assemblies.Add("IIEHost.dll");
assemblies.Add("Microsoft.Build.Conversion.dll");
assemblies.Add("Microsoft.Build.Engine.dll");
assemblies.Add("Microsoft.Build.Framework.dll");
assemblies.Add("Microsoft.Build.Tasks.dll");
assemblies.Add("Microsoft.Build.Utilities.dll");
assemblies.Add("Microsoft.Build.VisualJSharp.dll");
assemblies.Add("Microsoft.CompactFramework.Build.Tasks.dll");
assemblies.Add("Microsoft.JScript.dll");
assemblies.Add("Microsoft.VisualBasic.Compatibility.Data.dll");
assemblies.Add("Microsoft.VisualBasic.Compatibility.dll");
assemblies.Add("Microsoft.VisualBasic.dll");
assemblies.Add("Microsoft.VisualBasic.Vsa.dll");
assemblies.Add("Microsoft.Vsa.dll");
assemblies.Add("Microsoft.Vsa.Vb.CodeDOMProcessor.dll");
assemblies.Add("Microsoft_VsaVb.dll");
assemblies.Add("mscorlib.dll");
assemblies.Add("sysglobl.dll");
assemblies.Add("System.configuration.dll");
assemblies.Add("System.Configuration.Install.dll");
assemblies.Add("System.Data.dll");
assemblies.Add("System.Data.OracleClient.dll");
assemblies.Add("System.Data.SqlXml.dll");
assemblies.Add("System.Deployment.dll");
assemblies.Add("System.Design.dll");
assemblies.Add("System.DirectoryServices.dll");
assemblies.Add("System.DirectoryServices.Protocols.dll");
assemblies.Add("System.dll");
assemblies.Add("System.Drawing.Design.dll");
assemblies.Add("System.Drawing.dll");
assemblies.Add("System.EnterpriseServices.dll");
assemblies.Add("System.Management.dll");
assemblies.Add("System.Messaging.dll");
assemblies.Add("System.Runtime.Remoting.dll");
assemblies.Add("System.Runtime.Serialization.Formatters.Soap.dll");
assemblies.Add("System.Security.dll");
assemblies.Add("System.ServiceProcess.dll");
assemblies.Add("System.Transactions.dll");
assemblies.Add("System.Web.dll");
assemblies.Add("System.Web.Mobile.dll");
assemblies.Add("System.Web.RegularExpressions.dll");
assemblies.Add("System.Web.Services.dll");
assemblies.Add("System.Windows.Forms.dll");
assemblies.Add("System.XML.dll");
assemblies.Add("vjscor.dll");
assemblies.Add("vjsjbc.dll");
assemblies.Add("vjslib.dll");
assemblies.Add("vjslibcw.dll");
assemblies.Add("vjssupuilib.dll");
assemblies.Add("vjsvwaux.dll");
assemblies.Add("vjswfc.dll");
assemblies.Add("VJSWfcBrowserStubLib.dll");
assemblies.Add("vjswfccw.dll");
assemblies.Add("vjswfchtml.dll");
return;
}
}
}

Compile it into a separate assembly, send the assembly, have the other process load it.
You might want to consider security implications.
Update: another idea would be to generate an expression tree and use this library to serialize it:
http://www.codeplex.com/metalinq/

It is an interesting challenge, but you should probably describe why you want to do this, since there is a lot of different approaches depending on your objective. As humpohl points out, there is also some pretty serious security issues.
"Serialized code" could just be source code or a compiled assembly, depending on your requirements. You probably don't need to use a seperate code serialization format.
If you want to generate code dynamically and pass that on, you could generate code using CodeDOM and compile it. However, you most likely dont need to generate completely arbitrary code.

Another option is using the DLR, and constraining the code to execute...