We Keep Coding

What's the best way to load a CSX script from a C# application?

Tools like dotnet-script and CSI allow users to write, compile, and run C# like "scripts" rather than including their code in a complete pre-compiled project. These tools work great for command-line usage, but don't offer much in terms of integrating dynamic C# "scripts" into a larger C# application.
If I have an existing C# application which wishes to load additional classes into its existing namespaces via .csx "scripts", how do I do that? Is it possible?

I guess you need to compile and execute your C# script.
In my experience, I used C# scripts by referencing Microsoft.CodeAnalysis.CSharp.Scripting (version 3.*) directly.
<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Scripting" Version="3.*" />
Compilation
I suggest to use default compilation options:
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.CodeAnalysis.Scripting;
// ...
ScriptOptions options = ScriptOptions.Default;
Maybe in the future, you'll need to add referenced assemblies to your script compilation.
So you need to compile your script (contained in a string variable in the code below):
byte[] assemblyBinaryContent;
var roslynScript = CSharpScript.Create(script, options);
var compilation = roslynScript.GetCompilation();
compilation = compilation.WithOptions(compilation.Options
.WithOptimizationLevel(OptimizationLevel.Release)
.WithOutputKind(OutputKind.DynamicallyLinkedLibrary));
using (var assemblyStream = new MemoryStream())
{
var result = compilation.Emit(assemblyStream);
if (!result.Success)
{
var errors = string.Join(Environment.NewLine, result.Diagnostics.Select(x => x));
throw new Exception("Compilation errors: " + Environment.NewLine + errors);
}
assemblyBinaryContent = assemblyStream.ToArray();
}
GC.Collect(); // it allows to force clear compilation stuff.
Assembly assembly = Assembly.Load(assemblyBinaryContent);
var ret = Run(assembly); // see next paragraph
Execution
Obviously you need an entry point to execute your script.
I found out this trickly solution. It works.
private object Run(Assembly assembly)
{
//Execute the script
var type = assembly.GetType("Submission#0");
var method = type.GetMethod("<Factory>", BindingFlags.Static | BindingFlags.Public);
var retTask = method.Invoke(null, new object[] { new object[2] }) as Task<object>;
return retTask.GetAwaiter().GetResult();
}
I hope it can help.

.NET reference design for GUI app with built-in interactive terminal console (e.g. SublimeText, Visual Studio)

I'm trying to build a GUI app that has an interactive console, much like the one found in SublimeText.
I hope it is a valid question because it seems to be "a practical, answerable problem that is unique to software development".
In short, I see huge benefits having an interactive console inside a GUI app for
debugging, probing internal variables at runtime
logging
quick configuration changes
However, I have not come across any existing open-source applications that uses such a design.
I'm hoping someone has done it before and can share his/her design approach.
While I do have a semi-working solution using reflection and invoke in .NET, it is limited to only function calls and I'm not able to probe into nested internal variables (e.g. object.property.property).
To make the question more specific, these are the problems I'm facing:
Not easily extensible (Need to wire every new GUI command to a console command, vice-versa), any design tips? Routed commands (I could not find a useful example either)?
How to execute dynamic code that can access all existing object instances in the entire .NET app?
Thank you.

So here comes the code which worked for me:
namespace ReflectionsTest
{
public partial class MainForm : Form
{
public MainForm()
{
InitializeComponent();
}
//Events excluded
private void ExecuteCommand(string command)
{
string cmd = "";
cmd += #"using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.Linq;
using Microsoft.CSharp;
using System.Reflection;
using ReflectionsTest;";
// i included a using statement for every namespace i want to adress direct
cmd += #"namespace ReflectionConsole
{
public class RuntimeExecution
{
public static void Main(MainForm parent, TextBox output, FieldInfo[] privateFields)
{
try {";
//the code in a trycatch because i can send every error to a specific output defined as output parameter
cmd += command;
cmd += "}catch (Exception ex) { if(output != null){" +
"output.Text += ex.Message + \"\\n\\r\";"
+"}else{MessageBox.Show(ex.Message);}}}}}";
try {
ExecuteCSharp(cmd);
}
catch (Exception ex) {
textBox2.Text += ex.Message + "\n\r";
}
}
private void ExecuteCSharp(string code)
{
CSharpCodeProvider provider = new CSharpCodeProvider();
CompilerParameters parameters = new CompilerParameters();
List<AssemblyName> assemblys = (Assembly.GetExecutingAssembly().GetReferencedAssemblies()).ToList<AssemblyName>();
foreach (var item in assemblys) {
parameters.ReferencedAssemblies.Add(item.Name + ".dll");
}
string t = Assembly.GetExecutingAssembly().GetName().Name;
parameters.ReferencedAssemblies.Add(t + ".exe");
//Here you have to reference every assembly the console wants access
parameters.GenerateInMemory = true;
parameters.GenerateExecutable = false;
CompilerResults results = provider.CompileAssemblyFromSource(parameters, code);
if (results.Errors.HasErrors) {
StringBuilder sb = new StringBuilder();
foreach (CompilerError error in results.Errors) {
sb.AppendLine(String.Format("Error ({0}): {1}", error.ErrorNumber, error.ErrorText));
}
throw new InvalidOperationException(sb.ToString());
}
else {
Assembly assembly = results.CompiledAssembly;
Type program = assembly.GetType("ReflectionConsole.RuntimeExecution");
MethodInfo main = program.GetMethod("Main");
FieldInfo[] fields = this.GetType().GetFields(
BindingFlags.NonPublic |
BindingFlags.Instance);
//if everything is correct start the method with some arguments:
// containing class, output, private fields of the containing class for easier access
main.Invoke(null, new object[]{this, textBox2, fields});
}
}
}
}
Some Explanations:
You have pass the highest class of your program which contains everything else, because it is easier to access members than parent objects.
public objects you can access like parent.obect1.Text = "textXYZ";
private objects you can access by name. These objects are listed in privateFields.
for the subclasses you have two options:
change the first and third parameter when calling main.Invoke([...])
or
recollect the private fields.
as Suggestion you could include a .dll in the command which already gives you methods to achieve this much faster.
For example GetValueFromFieldByName(object class, string name, Type resultType)
I hope that is what you've hoped for ^^

Referencing current assembly with CompilerParameters

Right now I'm working on a project, and the team wants a way to write code and edit it without having to recompile the whole project, so I've decided to try and implement a scripting engine.
Having implemented Lua into C++ before, I wasn't an entire newbie to implementing scripting capabilities into projects. However, we wanted to try and implement straight C# using the Microsoft.CSharp namespace, combined with System.Reflection that was already built in to C#.
So having hearing about this, I poked about in docs and I've come up with a prototype that ALMOST works - but doesn't quite.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.CSharp;
using System.CodeDom.Compiler;
using System.Reflection;
namespace Scripting
{
class Program
{
static void Main(string[] args)
{
StringBuilder builder = new StringBuilder();
builder.Append("using System;");
builder.Append("using Scripting;");
builder.Append("class MyScript : IScript");
builder.Append("{");
builder.Append(" string ScriptName");
builder.Append(" {");
builder.Append(" get { return \"My Script\"; }");
builder.Append(" }");
builder.Append(" public bool Initialize()");
builder.Append(" {");
builder.Append(" Console.WriteLine(\"Hello, World!\");");
builder.Append(" return true;");
builder.Append(" }");
builder.Append("}");
CSharpCodeProvider provider = new CSharpCodeProvider();
CompilerParameters param = new CompilerParameters(new string[] { "System.dll", "Scripting.dll" });
param.GenerateInMemory = true;
param.GenerateExecutable = true;
CompilerResults result = provider.CompileAssemblyFromSource(param, builder.ToString());
if (result.Errors.Count > 0)
{
foreach (CompilerError error in result.Errors)
Console.WriteLine(error);
Console.ReadKey();
return;
}
}
}
}
The issue I have at the moment is that I want to be able to reference my interface - IScript.cs (which is inside the Scripting namespace and thus, the current assembly) - so that scripts written and parsed in the compiler can access it. Obviously, I added Scripting.dll as a parameter, but it doesn't seem to be able to be accessed for some reason or another. I am running it in debug so this could be cause for some major facepalmage. What do?
Is there a way to reference the current assembly and pass it to CompilerParameters? Or am I royally screwed / should I rely on creating an assembly for script objects / etc?

It's probably looking in the wrong directory.
Pass typeof(Program).Assembly.CodeBase to pass the full path.

You can get the executable and pass it to the CompilerParameters:
string exeName = Assembly.GetEntryAssembly().Location;
param.ReferencedAssemblies.Add(exeName);

.NET assemblies: understanding type visibility

I am trying to reproduce something that System.Xml.Serialization already does, but for a different source of data.
For now task is limited to deserialization only.
I.e. given defined source of data that I know how to read. Write a library that takes a random type, learns about it fields/properties via reflection, then generates and compiles "reader" class that can take data source and an instance of that random type and writes from data source into the object's fields/properties.
here is a simplified extract from my ReflectionHelper class
public class ReflectionHelper
{
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, XDataReader reader);
}
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = GetCSharpName(t);
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] {source});
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<EntityField> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + #".Generated
{
public class " + readerClassName + #" : ReflectionHelper.FieldReader<" + className + #">
{
public void Fill(" + className + #" e, XDataReader reader)
{
" + w.GetStringBuilder().ToString() + #"
}
}
}
";
}
}
and the calling code:
class Program
{
static void Main(string[] args)
{
ReflectionHelper.GetFieldReader<Foo>();
Console.ReadKey(true);
}
private class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
}
The dynamic compilation of course fails because Foo class is not visible outside of Program class. But! The .NET XML deserializer somehow works around that - and the question is: How?
After an hour of digging System.Xml.Serialization via Reflector I came to accept that I lack some kind of basic knowledge here and not really sure what am I looking for...
Also it is entirely possible that I am reinventing a wheel and/or digging in a wrong direction, in which case please do speak up!

You don’t need to create a dynamic assembly and dynamically compile code in order to deserialise an object. XmlSerializer does not do that either — it uses the Reflection API, in particular it uses the following simple concepts:
Retrieving the set of fields from any type
Reflection provides the GetFields() method for this purpose:
foreach (var field in myType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
// ...
I’m including the BindingFlags parameter here to ensure that it will include non-public fields, because otherwise it will return only public ones by default.
Setting the value of a field in any type
Reflection provides the function SetValue() for this purpose. You call this on a FieldInfo instance (which is returned from GetFields() above) and give it the instance in which you want to change the value of that field, and the value to set it to:
field.SetValue(myObject, myValue);
This is basically equivalent to myObject.Field = myValue;, except of course that the field is identified at runtime instead of compile-time.
Putting it all together
Here is a simple example. Notice you need to extend this further to work with more complex types such as arrays, for example.
public static T Deserialize<T>(XDataReader dataReader) where T : new()
{
return (T) deserialize(typeof(T), dataReader);
}
private static object deserialize(Type t, XDataReader dataReader)
{
// Handle the basic, built-in types
if (t == typeof(string))
return dataReader.ReadString();
// etc. for int and all the basic types
// Looks like the type t is not built-in, so assume it’s a class.
// Create an instance of the class
object result = Activator.CreateInstance(t);
// Iterate through the fields and recursively deserialize each
foreach (var field in t.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
field.SetValue(result, deserialize(field.FieldType, dataReader));
return result;
}
Notice I had to make some assumptions about XDataReader, most notably that it can just read a string like that. I’m sure you’ll be able to change it so that it works with your particular reader class.
Once you’ve extended this to support all the types you need (including int? in your example class), you can deserialize an object by calling:
Foo myFoo = Deserialize<Foo>(myDataReader);
and you can do this even when Foo is a private type as it is in your example.

If I try to use sgen.exe (the standalone XML serialization assembly compiler), I get the following error message:
Warning: Ignoring 'TestApp.Program'.
- TestApp.Program is inaccessible due to its protection level. Only public types can be processed.
Warning: Ignoring 'TestApp.Program+Foo'.
- TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
Assembly 'c:\...\TestApp\bin\debug\TestApp.exe' does not contain any types that can be serialized using XmlSerializer.
Calling new XmlSerializer(typeof(Foo)) in your example code results in:
System.InvalidOperationException: TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
So what gave you the idea that XmlSerializer can handle this?
However, remember that at runtime, there are no such restrictions. Trusted code using reflection is free to ignore access modifiers. This is what .NET binary serialization is doing.
For example, if you generate IL code at runtime using DynamicMethod, then you can pass skipVisibility = true to avoid any checks for visibility of fields/classes.

I've been working a bit on this. I'm not sure if it will help but, anyway I think it could be the way. Recently I worked with Serialization and DeSerealization of a class I had to send over the network. As there were two different programs (the client and the server), at first I implemented the class in both sources and then used serialization. It failed as the .Net told me it had not the same ID (I'm not sure but it was some sort of assembly id).
Well, after googling a bit I found that it was because the serialized class was on different assemblies, so the solution was to put that class in a independent library and then compile both client and server with that library. I've used the same idea with your code, so I put both Foo class and FieldReader class in a independent library, let's say:
namespace FooLibrary
{
public class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, IDataReader reader);
}
}
compile it and add it to the other source (using FooLibrary;)
this is the code I've used. It's not exactly the same as yours, as I don't have the code for GetCSharpName (I used t.Name instead) and XDataReader, so I used IDataReader (just for the compiler to accept the code and compile it) and also change EntityField for object
public class ReflectionHelper
{
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = t.Name;
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
object[] fields = new object[10];
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
prms.ReferencedAssemblies.Add("FooLibrary1.dll");
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] { source });
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<object> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + ".Generated
{
public class " + readerClassName + #" : FieldReader<" + className + #">
{
public override void Fill(" + className + #" e, IDataReader reader)
" + w.GetStringBuilder().ToString() +
}
}";
}
}
by the way, I found a tiny mistake, you should use new or override with the Fill method, as it is abstract.
Well, I must admit that GetFieldReader returns null, but at least the compiler compiles it.
Hope that this will help you or at least it guides you to the good answer
regards

FileVersionInfo.GetVersionInfo() incorrect in Console Application

I'm getting some serious weirdness using FileVersionInfo.GetVersionInfo() and was hoping somebody might be able to help.
The basics of the issue is that I am iterating through all the files in a folder calling GetVersionInfo() on each. There are about 300 files. This works ok for all but 2 of the files. For these DLLs I am getting comepletely incorrect info back from GetVersionInfo().
In order to eliminate all other variables, I extracted this call into a simple test app and it still got the same problem. However, if I built the test app as a Windows Application (it was a Console Application initially) then the data came back correct.
Just to clarify, the incorrect data coming back when running as a Console App is not simply null info like you would get if the file didn't contain version data. It contained reasonable data, but just the wrong data. It's as if it's reading it from a different file. I've looked for a file that contains matching version data, but can't find one.
Why is this simple call functioning differently if built as a Console Application rather than a Windows Application?
If anyone can help with this I would be very grateful.
Rgds,
Andy
-- Code Added
using System;
using System.Diagnostics;
namespace test
{
class Program
{
static void Main(string[] args)
{
string file = "C:\\ProblemFile.dll";
FileVersionInfo version = FileVersionInfo.GetVersionInfo(file);
string fileName = version.FileName;
string fileVersion = version.FileVersion;
Console.WriteLine(string.Format("{0} : {1}", fileName, fileVersion));
}
}
}

This behaviour seems weird indeed. Could it be that the Console application does not load the DLL from the same place as the WinForms application does? This would mean that GetVersionInfo uses some other API than just Win32 CreateFile (maybe going through some DLL resolver mechanism, side-by-side or whatever); remember that under the covers, version.dll will be executing your request, not the CLR itself.
Looking at FileVersionInfo through Reflector points in another direction yet:
public static unsafe FileVersionInfo GetVersionInfo(string fileName)
{
// ...
int fileVersionInfoSize = UnsafeNativeMethods.GetFileVersionInfoSize(fileName, out num);
FileVersionInfo info = new FileVersionInfo(fileName);
if (fileVersionInfoSize != 0)
{
byte[] buffer = new byte[fileVersionInfoSize];
fixed (byte* numRef = buffer)
{
IntPtr handle = new IntPtr((void*) numRef);
if (!UnsafeNativeMethods.GetFileVersionInfo(fileName, 0, fileVersionInfoSize, new HandleRef(null, handle)))
{
return info;
}
int varEntry = GetVarEntry(handle);
if (!info.GetVersionInfoForCodePage(handle, ConvertTo8DigitHex(varEntry)))
{
int[] numArray = new int[] { 0x40904b0, 0x40904e4, 0x4090000 };
foreach (int num4 in numArray)
{
if ((num4 != varEntry) && info.GetVersionInfoForCodePage(handle, ConvertTo8DigitHex(num4)))
{
return info;
}
}
}
}
}
return info;
}
As you can see there, some interesting dance is going on with code pages. What if the DLLs you inspected had several version information resources attached to them? Depending on the culture of the program calling into GetVersionInfo, I guess that the code page related calls could return other results?
Take the time to check the resources of the DLLs and make sure that there is only one language/code page for the version information. It might point you at the solution, I hope.

Sure the "files" you're seeing aren't . and .. ? If you iterate through all files, you'll always see entries for . (current dir) and .. (up dir). GetVersion Info might well return anything for these. You'd have to filter these entries out manually by name.

File and Assembly versions are 2 different things.
Are you sure you are not expecting the other?

Update: Tried this. Didn't work.
using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
namespace test
{
class Program
{
[DllImport("COMCTL32")]
private static extern int InitCommonControls(int nExitCode);
static void Main(string[] args)
{
InitCommonControls(0);
string file = "C:\\ProblemFile.dll";
FileVersionInfo version = FileVersionInfo.GetVersionInfo(file);
string fileName = version.FileName;
string fileVersion = version.FileVersion;
Console.WriteLine(string.Format("{0} : {1}", fileName, fileVersion));
}
}
}