We Keep Coding

Executing the code stored in string field of XML file

So, I'm making myself a small C# library for dialogues to use in an cRPG game.
The idea is, that the Dialogue object and it's fields (like DialogueNode and DialogueOption objects) are created based on an XML file, which I aim to make as simple as possible. The fields, except for lists or objects of types contained within the library, are - at best - string identifiers, to be acquired and parsed by outside means when needed.
I've basic funcionality implemented - XML serialization, running through the dialogue and exiting it - as well as basic, console based application interpreter and a WPF editor to create the dialogues, because writing the dialogue in plain XML is not the most comfortable thing in the world. (the last two are meant to be as much independent from the library as possible, except maybe for implementing what's inside to show/create)
All that being said, I've encountered a problem (actually two, the other one I'll cover in different question when I've the time after my exams).
After giving it some of my unexperienced 'noobish' thought, I've come to think, that I'd like to have some basic predicates stored either in my nodes or options - they would be later checked in game to determine, whether to display the node/option or leave it be (or whatever meddling with those to be honest). For example - an option is displayed if the player character have item X in his inventory, or the node is displayed when player has a certain minimum value of an attribute.
My idea of implementing is so far like that:
Having a field PredicateScript in an object
Having a bool method, that would be executed in runtime by the interpreter like this:
public bool DisplayPredicate(string predicateCode)
{
bool result = FunctionExecutingCSharpCode(predicateCode);
return result;
}
I've read some topics about compiling on the fly very brielfy, but I'm not sure if it's exactly what I want - I'm not sure how it would affect the performance of application (either the interpreter or the game itself), if it would be recompiled every few seconds...
I'm not pasting any code of what I'm trying to do, because either I'm yet to do this (as I'm not writing the code I'm not sure it will work) or it's the library structure which I don't think would be of relevance aside from what I explained I aim to do. ^^
Thanks. ^^

Given the standard .NET framework, there's no such thing as a C# "interpreter" you can feed code that gets executed. You could try to dynamically create C# code and have that compiled on the fly into an in-memory assembly, which you can then use using the .NET compiler services.
And example of this is given here: http://www.codeproject.com/Tips/715891/Compiling-Csharp-Code-at-Runtime.

Should I compile C# at runtime in this scenario?

We have a MS SQL 2005 database. The data is uploaded in this database from Excel sheets. These Excel sheets come from various vendors and most of them never maintain a predefined structure so that we can create a separate insertion/update module.
Until now I have created a module which accepts an Excel sheet in a predefined format and uploads the data from it. For every vendor someone has to format it in the predefined manner manually which takes too long and is a donkey work.
Now we have removed the manual formatting and put all database columns in one column, and drop-downs containing all Excel columns in another column. The user maps the desired columns and updates the database.
But most of the time this simple mapping falls short, since mostly we have to split/combine values based on some logic. To make it more user friendly and since most of my users are themselves C# programmers I have decided to remove the drop-downs and use text boxes where users can enter direct logic. For example:
dbcol1 = excel[col1].ToString().Replace("-"," ")
+(int.Parse( excel[col6].ToString())*.1).ToString();
While searching for this I came across dynamically compiling C# code. Since I have never done so before and also I am not sure that this method will be efficient I need to know that am I moving in the right direction with this. Or is there a more efficient and faster way to do this?

Script it up? C# Script is pretty handy, if you want to compile and run code on the fly;
If you can string format something that is correct C# it'll let you compile and run it.
http://www.csscript.net/
For example:
dynamic script = CSScript.Evaluator
.LoadMethod(#"void SayHello(string greeting)
{
Console.WriteLine(greeting);
}");
script.SayHello("Hello World!");
Probably not great for production, but you can at least prototype your idea

Normally you would use Reflection.Emit to generate a .NET assembly on the fly and execute that. However, it accepts only Intermediate Language instructions, not C#. And creating a whole new assembly each time you want to execute a piece of code is not efficient at all.
Roslyn is Microsoft's attempt at a compiler as a service. This is a new rewritten compiler for C# and VB.Net that can be called by an application to compile code, and it provides information about the code it is compiling.
Anders Hejlsberg (I believe) showed an example using the Roslyn compiler to implement a C# scripting interface: type some C# and it gets executed on the fly. While Roslyn is still in development, you might want to see if it would meed your needs by trying the Roslyn CTP.
C# as a Scripting Language in Your .NET Applications Using Roslyn is a CodeProject article that goes into some detail.
Finally, if you cannot (yet) or don't want to use Roslyn, or if it does not provide the execution speed you were looking for, then I think your best bet (in terms of performance) is to parse and interpret the scripting code yourself. However, this is error-prone hard work.

C++/C# Python Hook

I've searched good and Stack Overflow but couldn't find an answer to what I was looking for. Is there anyway to hook the call of Python functions from within C++/C#? Capture the function call plus it parameters?
Edit with an example:
def a(pOne):
//do stuff
def b():
a("a")
So on the call to 'a' I want C++ to hook that function (assuming the function name is known) and execute a C++ function/event on the call to 'a' with the parameters passed to the C++ function/event being the value of what was passed in for 'pOne'.
Thanks

There a couple of different options for C++ depending on what you want to do.
The best way to do what you want to do is to 'extend' Python.
My Favorite way to do this is Boost.Python
You can also use the raw C-API though I wouldn't recommend this.
For some examples of embedding and extending with Boost.Python you can look at this project I have been working on.

If you just want to write some C++ code and make it callable from Python, you do that by Extending and Embedding the Python Interpreter. (Actually, just Extending; ignore the other half.)
Basically, if you write a new module foo, anyone can import foo and call foo.a("a"), and it doesn't matter if module foo is implemented as a Python file foo.py, or compiled from foo.cpp into a dynamic library foo.so.
For that, as Kyle C suggests, there are a number of higher-level ways to do this so you don't need to deal with the C API directly. (In addition to his suggestion of Boost.Python, I'd also suggest looking at Cython, which lets you write the code in an almost-Python language that can talk directly to your C++, while also exposing things directly to Python.)
However, that isn't what you asked for. What you want to be able to do is take some function that's defined in Python code, and hook it to do something different. For that, you really do need to read the Extending and Embedding documentation linked above. You're going to have to write an embedded interpreter, reproduce some of the behavior (exactly how much depends on where exactly you want to hook it), and make sure that wherever PyObject_Call or a similar function would have been called, it first checks whether that object is the a function and, if so, calls your hook code.
This is going to be pretty difficult. If you haven't written an embedded Python interpreter yet, go do that before you even think about how to hook it.
It's worth noting that it's probably much, much easier to do the hooking from within Python than from outside the interpreter. (If you've never heard of "monkeypatching", go google that.) And you can always make your Python hook call code from a module that you built in C++, or even call directly into a compiled .so file via ctypes.
Finally, if you want to hook some running interpreter instance at runtime, that's even more difficult. You obviously need to be able to do some kind of debug/trace/etc. attach and code insertion, and the details of that are entirely dependent on your platform. Then, you'll want to do the same thing you would have done in the previous hard version, except that you'll have to do it by intercepting calls to, e.g., PyObject_Call (from libpython.so/Python.dll/Python.framework/whatever) to go through your hooks first. If you don't already know how to hook SO calls on your platform, you need to learn that before you can even think about hooking Python code from outside.

Do you have reasons not to alter the in-python code? If not, why not write an extension module or a python ctypes call to your dll, and wrap a with a decorator that calls your hook?
import my_cpp_module
def hook_a_to_call_my_cpp_module(orig_a):
def replacement_a(pOne):
try:
my_cpp_module.my_cpp_function(pOne)
finally:
return orig_a(pOne)
#
return replacement_a
#hook_a_to_call_my_cpp_module
def a(pOne):
pass
you can even do this without touching the source file which has a, like this:
# in this other file you import the module that contains function a
import the_py_that_has_a
# next line just to avoid copy-pasting the above code from my answer here again
from the_decorator_from_the_previous_example import hook_a_to_call_my_cpp_module
the_py_that_has_a.a = hook_a_to_call_my_cpp_module(the_py_that_has_a.a)
If you have some reasons not to alter your py code at all, could you state them in a comment?

Make an executable at runtime

Ok, so I was wondering how one would go about creating a program, that creates a second program(Like how most compression programs can create self extracting self excutables, but that's not what I need).
Say I have 2 programs. Each one containing a class. The one program I would use to modify and fill the class with data. The second file would be a program that also had the class, but empty, and it's only purpose is to access this data in a specific way. I don't know, I'm thinking if the specific class were serialized and then "injected" into the second file. But how would one be able to do that? I've found modifying files that were already compiled fascinating, though I've never been able to make changes that didn't cause errors.
That's just a thought. I don't know what the solution would be, that's just something that crossed my mind.
I'd prefer some information in say c or c++ that's cross-platform. The only other language I'd accept is c#.
also
I'm not looking for 3-rd party library's, or things such as Boost. If anything a shove in the right direction could be all I need.
++also
I don't want to be using a compiler.
Jalf actually read what I wrote
That's exactly what I would like to know how to do. I think that's fairly obvious by what I asked above. I said nothing about compiling the files, or scripting.
QUOTE "I've found modifying files that were already compiled fascinating"
Please read and understand the question first before posting.
thanks.

Building an executable from scratch is hard. First, you'd need to generate machine code for what the program would do, and then you need to encapsulate such code in an executable file. That's overkill unless you want to write a compiler for a language.
These utilities that generate a self-extracting executable don't really make the executable from scratch. They have the executable pre-generated, and the data file is just appended to the end of it. Since the Windows executable format allows you to put data at the end of the file, caring only for the "real executable" part (the exe header tells how big it is - the rest is ignored).
For instance, try to generate two self-extracting zip, and do a binary diff on them. You'll see their first X KBytes are exactly the same, what changes is the rest, which is not an executable at all, it's just data. When the file is executed, it looks what is found at the end of the file (the data) and unzips it.
Take a look at the wikipedia entry, go to the external links section to dig deeper:
http://en.wikipedia.org/wiki/Portable_Executable
I only mentioned Windows here but the same principles apply to Linux. But don't expect to have cross-platform results, you'll have to re-implement it to each platform. I couldn't imagine something that's more platform-dependent than the executable file. Even if you use C# you'll have to generate the native stub, which is different if you're running on Windows (under .net) or Linux (under Mono).

Invoke a compiler with data generated by your program (write temp files to disk if necessary) and or stored on disk?
Or is the question about the details of writing the local executable format?

Unfortunately with compiled languages such as C, C++, Java, or C#, you won't be able to just ``run'' new code at runtime, like you can do in interpreted languages like PHP, Perl, and ECMAscript. The code has to be compiled first, and for that you will need a compiler. There's no getting around this.
If you need to duplicate the save/restore functionality between two separate EXEs, then your best bet is to create a static library shared between the two programs, or a DLL shared between the two programs. That way, you write that code once and it's able to be used by as many programs as you want.
On the other hand, if you're really running into a scenario like this, my main question is, What are you trying to accomplish with this? Even in languages that support things like eval(), self modifying code is usually some of the nastiest and bug-riddled stuff you're going to find. It's worse even than a program written completely with GOTOs. There are uses for self modifying code like this, but 99% of the time it's the wrong approach to take.
Hope that helps :)

I had the same problem and I think that this solves all problems.
You can put there whatever code and if correct it will produce at runtime second executable.
--ADD--
So in short you have some code which you can hard-code and store in the code of your 1st exe file or let outside it. Then you run it and you compile the aforementioned code. If eveything is ok you will get a second executable runtime- compiled. All this without any external lib!!

Ok, so I was wondering how one would
go about creating a program, that
creates a second program
You can look at CodeDom. Here is a tutorial

Have you considered embedding a scripting language such as Lua or Python into your app? This will give you the ability to dynamically generate and execute code at runtime.
From wikipedia:
Dynamic programming language is a term used broadly in computer science to describe a class of high-level programming languages that execute at runtime many common behaviors that other languages might perform during compilation, if at all. These behaviors could include extension of the program, by adding new code, by extending objects and definitions, or by modifying the type system, all during program execution. These behaviors can be emulated in nearly any language of sufficient complexity, but dynamic languages provide direct tools to make use of them.

Depending on what you call a program, Self-modifying code may do the trick.
Basically, you write code somewhere in memory as if it were plain data, and you call it.
Usually it's a bad idea, but it's quite fun.

Where would you use C# Runtime Compilation?

I happened upon a brief discussion recently on another site about C# runtime compilation recently while searching for something else and thought the idea was interesting. Have you ever used this? I'm trying to determine how/when one might use this and what problem it solves. I'd be very interested in hearing how you've used it or in what context it makes sense.
Thanks much.

Typically, I see this used in cases where you are currently using Reflection and need to optimize for performance.
For example, instead of using reflection to call method X, you generate a Dynamic Method at runtime to do this for you.

You can use this to add scripting support to your application. For examples look here or here.
It is quite easily possible to publish parts of your internal object framework to the scripting part, so you could with relative ease add something to your application that has the same effect as for example VBA for Office.

I've seen this (runtime compilation / use of System.Reflection.Emit classes) in generating dynamic proxies ( Code project sample ) or other means of optimizing reflection calls (time-wise).

At least one case you might use it is when generating dynamic code. For example, the framework is using this internally to generate XML serializers on the fly. After looking into a class at runtime, it can generate the code to serialize / deserialize the class. It then compiles that code and users it as needed.
In the same way you can generate code to handle arbitrary DB tables etc. and then compile and load the generated assembly.

Well, all C# code is run-time compiled, since it's a JIT (just-in-time) compiler. I assume you are referring to Reflection.Emit to create classes etc. on the fly. Here's an example I have seen recently in the Xml-Rpc.Net library.
I create a C# interface that has the same signature as an XML-RPC service's method calls, e.g.
IMyProxy : IXmlRpcProxy
{
[XmlRpcMethod]
int Add(int a, int b);
}
Then in my code I call something like
IMyProxy proxy = (IMyProxy)XmlRcpFactory.Create(typeof(IMyProxy));
This uses run-time code generation to create a fully functional proxy for me, so I can use it like this:
int result = proxy.Add(1, 2);
This then handles the XML-RPC call for me. Pretty cool.

I used runtime compiler services from .NET in my diploma thesis. Basically, it was about visually creating some graphical component for a process visualization, which is generated as C# code, compiled into an assembly and can then be used on the target system without being interpreted, to make it faster and more compact. And, as a bonus, the generated images could be packaged into the very same assembly as resources.
The other use of that was in Java. I had an application that had to plot a potentially expensive function using some numerical algorithm (was back at university) the user could enter. I put the entered function into a class, compiled and loaded it and it was then available for relatively fast execution.
So, these are my two experiences where runtime code generation was a good thing.

something I used it for was for allowing C# and VB code to bu run by the user ad-hoc. They could type in a line of code (or a couple lines) and it would be compiled, loaded into an app domain, and executed, and then unloaded. This probably isnt the best example of its usage, but an example of it none-the-less