I'm thinking of writing a program that involves including super fast Assembly or as it dosn't have to be human readable it could be Machine Code in C++ or C#. However I also have other possibly more troublesome requirements.
I would need to be able to:
Store machine code programs in normal variables / object instances, for example strings "40 9B 7F 5F ..." to edit and run them.
Have the programs able to output data. I saw an example where one had a pointer to an int that it could use.
Have the programs not able to output data anywhere else. For example to not be able to perform such actions as to delete files, view the system spec or change the state of the memory of the C++ or C# program they are contained within.
For example, it could be something like this:
machine n;
n = "40 9B 7F";
n[1] = "5F";
// 'n' is now "40 5F 7F"
unsigned short s = 2;
n.run(&s);
// while 'n' was running it may have changed 's' but would not have been able to
// change anything else anywhere on the system including in this C++ / C# program
According to the wiki link Michael Dorgan posted "asm(std::string);" runs the String as assembler and it's also easy to referance variables from the C++ part of the program. Editing a std::String is easy and Alex has noted that I can ensure that the code is safe by not allowing unsafe commands.
Sandboxing native machine code is non-trivial. If you really want that take a look at NACL from google which implements a machine code sandbox for browsers.
What is more practical is to use .NET IL instead of machine code and use a sandboxed (or hosted) AppDomain. This comes much closer and still is fast due to the dynamically jit-compilation to machine code.
An alternative you have is to use Windows builtin rights management and spawn a new process with restricted rights. Never done that so I don't know if you can reduce the target processes rights as much as you want. Anyways that would be a pure win32 process just running machine code, so you lose any ability of using .NET in the sandboxed process.
If you want to include assembler in your C/C++ code, consider either inline assembly routines, or compiling seperate full on assembler files and linking them back in. Inline assembler syntax is kinda weird, but I believe it is probably the best choice for you from what I've read.
Wikipedia to the rescue for some samples:
Inline assembler examples
Update based on comments:
This is far from a trivial task. You have to implement a linker, assembler (to scan and sandbox) and loader.
I wonder what the use case is -- for my example I'll assume you want to to have an assembly contest where people submit solutions to problems and you "test" them.
This is the best solution I can think of:
Have a hosting program that takes as input assembly language.
Invoke the assembler to compile and link the assembly program.
Create a protected virtual environment for the program to run in (how you do this depends on the platform) which runs as a user that has no rights to the system.
Capture the results
This solution allows you to leverage existing assemblers, loaders and security without having to re-implement them.
The best example code of dynamically loading, running and sandboxing C# code I know of is the terrarium game at http://terrarium2.codeplex.com/
However, you might consider something better suited to this job, like a scripting system. Lua comes to mind as a popular one. Using Lua users will only be able to perform the actions you allow. http://www.lua.org/
If you restrict the subset of supported instructions, you can do what you want more or less easily.
First, you have to parse and decode an input instruction to see if it's in the supported subset (most of parsing/decoding can be done just once). Then you need to execute it.
But before executing, there's one important thing to take care of. Based on the decoded details of the instruction and the CPU registers state, you have to calculate the memory addresses that the instruction is going to access as data (including on-stack locations) or transfer control to. If any of those are outside of the established limits, fire alarm. Otherwise, if it's a control transferring instruction (e.g. jmp, jz), you must additionally ensure that the address it passes control to is not only within the memory, where all these instructions lie, but also is the address of one of those instructions and not an address inside of any of them (e.g. 1 or 2 bytes from the beginning of a 3+ bytes long instruction). Passing control anywhere else is a no-no. You do not want these instructions to pass control to any standard library functions either because you won't be able to control execution there and they're not always safe when supplied with bogus/malicious inputs. Also, these instructions must not be able to modify themselves.
If all is clear, you can either emulate the instruction or more or less directly execute it (control passing instructions will likely have to be always emulated because you want to stop execution after every instruction). For the latter you can create a modifiable function containing these things:
Code to save CPU registers of the caller and load them with the state for the instruction being executed.
The instruction.
The reverse of step 1: code to save post-execution register state and restore the caller's register state.
You can try this approach.
Related
Is there a way/system to debug/monitor code without stopping execution?
In industrial automation control programming (PLC/PAC/DCS) it is possible to connect the debugger while the program is running, and see in the code editor the value of variables and expressions, without setting breakpoints or tracepoints.
As an example, let's have a F# multithreaded application, where code is executed in a continuous loop or triggered by timers. Is there a way to attach a debugger like Visual studio Debugger and see the values of variables and expressions (in the code editor or in a watch pane) WITHOUT interrupting the execution?
It doesn't matter if it's not synchronous, it's acceptable if the debugger/monitor does not capture all the code scans.
I am tasked to create an high level controller for a process plant and I would like to use C# or F# or even C++ with a managed or native application, instead of a PAC system. But being forced to interrupt execution to debug is a huge disadvantage in this kind of application.
UPDATE
First of all thanks to all for their answer.
Based on those answers, though, I realized that probably I need to reformulate my question as follows:
Is anyone aware of any library/framework/package/extension that allows to work with a native or managed application in windows or linux (C#, F# or C++) the exact same way as a PAC development platform, specifically:
1) Put the dev platform in "status" mode, where it shows automatically the runtime value for variables and expressions present in the code exceprt currently visible, without interrupting execution?
2) Create watch windows that show the runtime value of variables and expressions, again without interrupting execution?
Also, what I am looking for is something that (like any PAC platform) offers these features OUT OF THE BOX, without requiring any change in the application code (like adding log instructions).
Thank you in advance
UPDATE 2
It looks like there is something (see http://vsdevaids.webs.com/); does anyone know whether they are still available somewhere?
UPDATE 3
For those interested, I managed to download the last available release of VSDEVAIDS. I installed it and looks working, but it's pointless without a licence and couldn't find information on how to reach the author.
http://www.mediafire.com/file/vvdk2e0g6091r4h/VSDevAidsInstaller.msi
If somebody has better luck, please let me know.
this is a normal requirement - needing instrumentation / diagnostic data from a production system. Its not really a debugger. Its usually one of the first things you should establish in your system design.
Not knowing your system at all its hard to say what you need but generally they fall into 2 categories
human readable trace - something like log4net is what I would recommend
machine readable counters etc. Say 'number of widget shaving in last pass',..... This one is harder to generalize, you could layer it onto log4net too. Or invent your own pipe
With regards to your edited question, I can almost guarantee you that what you are looking for does not exist. Consequence-free debugging/monitoring of even moderate usefulness for production code with no prior effort? I'd have heard of it. Consider that both C++ and C# are extremely cross-platform. There are a few caveats:
There are almost certainly C++ compilers built for very specific hardware that do what you require. This hardware is likely to have very limited capabilities, and the compilers are likely to otherwise be inferior to their larger counterparts, such as gcc, clang, MSVC, to name a few.
Compile-time instrumentation can do what you require, although it affects speed and memory usage, and even stability, in my experience.
There ARE also frameworks that do what you require, but not without affecting your code. For example, if you are using WPF as your UI, it's possible to monitor anything directly related to the UI of your application. But...that's hardly a better solution than log4net.
Lastly, there are tools that can monitor EVERY system call your application makes for both Windows (procmon.exe/"Process Monitor" from SysInternals) and Linux (strace). There's very little you can't find out using these. That said, the ease of use is hardly what you're looking for, and strictly internal variables are still not going to be visible. Still might be something to consider if you know you'll be making system calls with the variables you're interested in and can set up adequate filtering.
Also, you should reconsider your "No impact on the code" requirement. There are .NET frameworks that can allow you to monitor an entire class merely by making a single function call during construction, or by deriving from a class in the framework. Many modern UIs are predicated on the UIs being able to be notified of any change to the data they are monitoring. Extensive effort has gone into making this as powerful and easy as possible. But it does require you to at least consider it when writing your code.
Many years ago (think 8 bit 6502/6809 days) you could buy (or usually rent, I seem to remember a figure of £40K to purchase one in the late 80s) a processor simulator, that would allow you replace the processor in your design with a pin compatible device that had a flying lead to the simulator box. this would allow things like capturing instructions/data leading up to a processor interrupt, or some other way of stopping the processor (even a 'push button to stop code' was possible). You could even step-backwards allowing you to see why an instruction or branch happened.
In these days of multi-core, nm-technology, I doubt there is such a thing.
I have been searching for this kind of features since quite a long time with no luck, unfortunately. Submitting the question to the StackOverflow community was sort of a "last resort", so now I'm ready to conclude that it doesn't exist.
VSDevAids (as #zzxyz pointed out) is not a solution, as it requires significant support from the application itself.
Pod cpu emulators (mentioned by #Neil) aka in-circuit emulators (ICE) and their evolutions are designed to thoroughly test the interaction between firmware and hardware, not so useful in high level programming (especially if managed like .NET).
Thanks for all contributions.
I'm trying to check if an application tries to manipulate a particular process (for ex. hooks itself to it). I couldn't find a proper approach to accomplish this. Is computing checksum over running process possible? If it's not how can i detect this situation?
Other process can't make hooks in your process, can modify memory but to make hooks this code must be in your address space, this can be done to injecting DLL to your process when is starting (at runtime inject dll is a hard one), you can easy check this by listing DLL's in your process and searching some ReadProcessMemory, WriteProcessMemory, OpenProcess, CallNextHookEx functions calls in their code. To do that get address (GetProcAddress) of function and search value in code (you can add some asm call predictions for that for tight range result).
You can check what is wrong with your PE file in disk and in memory, when DLL injection at startup time was occurrence then your PE file after was copied to memory from file should be corrupted, after last dll library you should have overwritten debug symbols with additional dll import. This modification can be done on file same as in memory.
The best method but probably will not easy for you when you are using C# language is obfuscate your code. I think this is a good method because you don't hook something that you don't know how work, because you don't know what hook you must do and where. But for good obfuscate C# code you must find good software for that and probably pay not low price.
If I understand this correctly:
Current CPU developing companies like AMD and Intel have their own API codes (the assembly language) as what they see as the 2G language on top of the Machine code (1G language)
Would it be possible or desirable (performance or otherwise) to have a CPU that would perform IL handling at it's core instead of the current API calls?
A similar technology does exist for Java - ARM do a range of CPUs that can do this, they call it their "Jazelle" technology.
However, the operations represented by .net IL opcodes are only well-defined in combination with the type information held on the stack, not on their own. This is a major difference from Java bytecode, and would make it much more difficult to create sensible hardware to execute IL.
Moreover, IL is intended for compilation to a final target. Most back ends that spit out IL do very little optimisation, aiming instead to preserve semantic content for verification and optimisation in the final compilation step. Even if the hardware problems could be overcome, the result will almost certainly still be slower than a decent optimising JIT.
So, to sum up: while it is not impossible, it would be disproportionately hard compared to other architectures, and would achieve little.
You seem a bit confused about how CPU's work. Assembly is not a separate language from machine code. It is simply a different (textual) representation of it.
Assembly code is simply a sequential listing of instructions to be executed. And machine code is exactly the same thing. Every instruction supported by the CPU has a certain bit-pattern that cause it to be executed, and it also has a textual name you can use in assembly code.
If I write add $10, $9, $8 and run it through an assembler, I get the machine code for the add instruction, taking the values in registers 9 and 8, adding them and storing the result in register 10.
There is a 1 to 1 mapping between assembler and machine code.
There also are no "API calls". The CPU simply reads from address X, and matches the subsequent bits against all the instructions it understands. Once it finds an instruction that matches this bit pattern, it executes the instruction, and moves on to read the next one.
What you're asking is in a sense impossible or a contradiction. IL stands for Intermediate Language, that is, a kind of pseudocode that is emitted by the compiler, but has not yet been translated into machine code. But if the CPU could execute that directly, then it would no longer be intermediate, it would be machine code.
So the question becomes "is your IL code a better, more efficient representation of a program, than the machine code the CPU supports now?"
And the answer is most likely no. MSIL (I assume that's what you mean by IL, which is a much more general term) is designed to be portable, simple and consistent. Every .NET language compiles to MSIL, and every MSIL program must be able to be translated into machine code for any CPU anywhere. That means MSIL must be general and abstract and not make assumptions about the CPU. For this reason, as far as I know, it is a purely stack-based architecture. Instead of keeping data in registers, each instruction processes the data on the top of the stack. That's a nice clean and generic system, but it's not very efficient, and doesn't translate well to the rigid structure of a CPU. (In your wonderful little high-level world, you can pretend that the stack can grow freely. For the CPU to get fast access to it, it must be stored in some small, fast on-chip memory with finite size. So what happens if your program push too much data on the stack?)
Yes, you could make a CPU to execute MSIL directly, but what would you gain?
You'd no longer need to JIT code before execution, so the first time you start a program, it would launch a bit faster. Apart from that, though? Once your MSIL program has been JIT'ed, it has been translated to machine code and runs as efficiently as if it had been written in machine code originally. MSIL bytecode no longer exists, just a series of instructions understood by the CPU.
In fact, you'd be back where you were before .NET. Non-managed languages are compiled straight to machine code, just like this would be in your suggestion. The only difference is that non-managed code targets machine code that is designed by CPU designers to be suitable for execution on a CPU, while in your case, it'd target machine code that's designed by software designers to be easy to translate to and from.
This is not a new idea - the same thing was predicted for Java, and Lisp machines were even actually implemented.
But experience with those shows that it's not really useful - by designing special-purpose CPUs, you can't benefit from the advances of "traditional" CPUs, and you very likely can't beat Intel at their own game. A quote from the Wikipedia article illustrates this nicely:
cheaper desktop PCs soon were able to
run Lisp programs even faster than
Lisp machines, without the use of
special purpose hardware.
Translating from one kind of machine code to another on the fly is a well-understood problem and so common (modern CISC CPUs even do something like that internally because they're really RISC) that I believe we can assume it is being done efficiently enough that avoiding it does not yield significant benefits - not when it means you have to decouple yourself from the state of the art in traditional CPUs.
I would say no.
The actual machine language instructions that need to run on a computer are lower level than IL. IL, for example, doesn't really describe how methods calls should be made, how registers should be managed, how the stack should be accessed, or any other of the details that are needed at the machine code level.
Getting the machine to recognize IL directly would, therefore, simple move all the JIT compilation logic from software into hardware.
That would make the whole process very rigid and unchangeable.
By having the machine language based on the capabilities of the machine, and an intermediate language based on capturing programmer intent, you get a much better system. The folks defining the machine can concentrate on defining an efficient computer architecture, and the folks defining the IL system can focus on things like expressiveness and safety.
If both the tool vendors and the hardware vendors had to use the exact same representation for everything, then innovation in either the hardware space or the tool space would be hampered. So, I say they should be separate from one another.
I wouldn't have thought so for two reasons:-
If you had hardware processing IL that hardware would not be able to run a newer version of IL. With JIT you just need a new JIT then existing hardware can run the newer IL.
IL is simple and is designed to be hardware agnostic. The IL would have to become much more complex to enable it to describe operations in the most effecient manner in order to get anywhere close to the performance of existing machine code. But that would mean the IL would be much harder to run on non IL specific hardware.
I'm writing my own scripting language in C#, with some features I like, and I chose to use MSIL as output's bytecode (Reflection.Emit is quite useful, and I dont have to think up another bytecode). It works, emits executable, which can be run ( even decompiled with Reflector :) )and is quite fast.
But - I want to run multiple 'processes' in one process+one thread, and control their assigned CPU time manually (also implement much more robust IPC that is offered by .NET framework) Is there any way to entirely disable JIT and create own VM, stepping instruction-after-instruction using .NET framework (and control memory usage, etc.), without need to write anything on my own, or to achieve this I must write entire MSIL interpret?
EDIT 1): I know that interpreting IL isn't the fastest thing in the universe :)
EDIT 2): To clarify - I want my VM to be some kind of 'operating system' - it gets some CPU time and divides it between processes, controls memory allocation for them, and so on. It doesnt have to be fast, nor effective, but just a proof of concept for some of my experiments. I dont need to implement it on the level of processing every instruction - if this should be done by .NET, I wont mind, i just want to say : step one instruction, and wait till I told you to step next.
EDIT 3): I realized, that ICorDebug can maybe accomplish my needs, now looking at implementation of Mono's runtime.
You could use Mono - I believe that allows an option to interpret the IL instead of JITting it. The fact that it's open source means (subject to licensing) that you should be able to modify it according to your needs, too.
Mono doesn't have all of .NET's functionality, admittedly - but it may do all you need.
Beware that MSIL was designed to be parsed by a JIT compiler. It is not very suitable for an interpreter. A good example is perhaps the ADD instruction. It is used to add a wide variety of value type values: byte, short, int32, int64, ushort, uint32, uint64. Your compiler knows what kind of add is required but you'll lose that type info when generating the MSIL.
Now you need to find it back at runtime and that requires checking the types of the values on the evaluation stack. Very slow.
An easily interpreted IL has dedicated ADD instructions like ADD8, ADD16, etc.
Microsofts implementation of the Common Language Runtime has only one execution system, the JIT. Mono, on the other hand comes with both, a JIT and an interpreter.
I, however, do not fully understand what exactly you want to do yourself and what you would like to leave to Microsofts implementation:
Is there any way to entirely disable JIT and create own VM?
and
... without need to write anything on my own, or to achieve this I must write entire MSIL interpret?
is sort of contradicting.
If you think, you can write a better execution system than microsofts JIT, you will have to write it from scratch. Bear in mind, however, that both microsofts and monos JIT are highly optimized compilers. (Programming language shootout)
Being able to schedule CPU time for operating system processes exactly is not possible from user mode. That's the operating systems task.
Some implementation of green threads might be an idea, but that is definitely a topic for unmanaged code. If that's what you want, have a look at the CLR hosting API.
I would suggest, you try to implement your language in CIL. After all, it gets compiled down to raw x86. If you don't care about verifiability, you can use pointers where necessary.
One thing you could consider doing is generating code in a state-machine style. Let me explain what I mean by this.
When you write generator methods in C# with yield return, the method is compiled into an inner IEnumerator class that implements a state machine. The method's code is compiled into logical blocks that are terminated with a yield return or yield break statement, and each block corresponds to a numbered state. Because each yield return must provide a value, each block ends by storing a value in a local field. The enumerator object, in order to generate its next value, calls a method that consists of a giant switch statement on the current state number in order to run the current block, then advances the state and returns the value of the local field.
Your scripting language could generate its methods in a similar style, where a method corresponds to a state machine object, and the VM allocates time by advancing the state machine during the time allotted. A few tricky parts to this method: implementing things like method calls and try/finally blocks are harder than generating straight-up MSIL.
I have a compiler which compiles assembly language to machine language (in memory).
My project is in C# .net.
Is there any way to run the memory on a thread?
How can DEP prevent it?
byte[] a:
01010101 10111010 00111010 10101011 ...
The key is to put the executable code into a block of memory allocated with VirtualAlloc such that the buffer is marked as executable.
IntPtr pExecutableBuffer = VirtualAlloc(
IntPtr.Zero,
new IntPtr(byteCount),
AllocationType.MEM_COMMIT | AllocationType.MEM_RESERVE,
MemoryProtection.PAGE_EXECUTE_READWRITE);
(then use VirtualFree to clean up after yourself).
This tells Windows that the memory should be marked as executable code so that it won't trigger a DEP check.
I doubt there's a supported way. I don't know and haven't researched it, but here are some guesses:
The easiest way might be to launch it as a process: write it into a *.com file and then tell the O/S to run that executable.
Alternatively, pass the memory as a parameter to the CreateThread function (but you'll need to wrorry about the code having the right calling conventions, expecting the specified parameters, preserving registers, and being in memory which is executable).
Another possibility is to write the opcodes into memory which is know is already going to be executed (e.g. overwrite existing code in a recently-loaded DLL).
It's possible to execute bytes as code:
Inline x86 ASM in C#
It does require the use of unsafe code.
I thought that this was just a fun fact but useless in practice, but perhaps your application actually has a use for this :)
You can whitelist your application from the control panel
http://ask-leo.com/how_do_i_turn_off_data_execution_prevention_errors.html
I doubt you can whitelist it programattically, but certainly not without admin access - that would defeat the purpose of this security feature.