Related
Question
How does a delegate store a reference to a function? The source code appears to refer to it as an Object, and the manner in which it invokes the method seems redacted from the source code. Can anyone explain how C# is handling this?
Original Post
It seems I'm constantly fighting the abstractions C# imposes on its programmers. One that's been irking me is the obfuscation of Functions/Methods. As I understand it, all methods are in fact anonymous methods assigned to properties of a class. This is the reason why no function is prefixed by a datatype. For example...
void foo() { ... }
... would be written in Javascript as...
Function foo = function():void { ... };
In my experience, Anonymous functions are typically bad form, but here it's replete throughout the language standard. Because you cannot define a function with its datatype (and apparently the implication/handling is assumed by the compiler), how does one store a reference to a method if the type is never declared?
I'm trying very hard to avoid Delegates and its variants (Action & Func), both because...
it is another abstraction from what's actually happening
the unnecessary overhead required to instantiate these classes (which in turn carry their own pointers to the methods being called).
Looking at the source code for the Delegate.cs, it appears to refer to the reference of a function as simply Object (see lines 23-25).
If these really are objects, how are we calling them? According to the delegate.cs trail, it dead-ends on the following path:
Delegate.cs:DynamicInvoke() > DynamicInvokeImpl() > methodinfo.cs:UnsafeInvoke() > UnsafeInvokeInternal() > RuntimeMethodHandle.InvokeMethod() > runtimehandles.cs:InvokeMethod()
internal extern static object InvokeMethod(object target, object[] arguments, Signature sig, bool constructor);
This really doesn't explain how its invoked if indeed the method is an object. It feels as though this is not code at all, and the actual code called has been redacted from source repository.
Your help is appreciated.
Response to Previous Comments
#Amy: I gave an example immediately after that statement to explain what I meant. If a function were prefixed by a datatype, you could write a true anonymous function, and store it as a property to an Object such as:
private Dictionary<string, Function> ops = new Dictionary<string, Function> {
{"foo", int (int a, int b) { return a + b } }
};
As it stands, C# doesn't allow you to write true anonymous functions, and walls that functionality off behind Delegates and Lambda expressions.
#500 Internal server error: I already explained what I was trying to do. I even bolded it. You assume there's any ulterior motive here; I'm simply trying to understand how C# stores a reference to a method. I even provided links to the source code so that others could read the code for themselves and help answer the question.
#Dialecticus: Obviously if I already found the typical answer on Google, the only other place to find the answer I'm looking for would be here. I realize this is outside the knowledge of most C# developers, and that's why I've provided the source code links. You don't have to reply if you don't know the answer.
While I'm not fully understanding your insights about "true anonymous functions", "not prefixed by a data type" etc, I can explain you how applications written in C# call methods.
First of all, there is no such a thing "function" in C#. Each and every executable entity in C# is in fact a method, that means, it belongs to a class. Even if you define lambdas or anonymous functions like this:
collection.Where(item => item > 0);
the C# compiler creates a compiler-generated class behind the scenes and puts the lambda body return item > 0 into a compiler-generated method.
So assuming you have this code:
class Example
{
public static void StaticMethod() { }
public void InstanceMethod() { }
public Action Property { get; } = () => { };
}
static class Program
{
static void Main()
{
Example.StaticMethod();
var ex = new Example();
ex.InstanceMethod();
ex.Property();
}
}
The C# compiler will create an IL code out of that. The IL code is not executable right away, it needs to be run in a virtual machine.
The IL code will contain a class Example with two methods (actually, four - a default constructor and the property getter method will be automatically generated) and a compiler-generated class containing a method whose body is the body of the lambda expression.
The IL code of Main will look like this (simplified):
call void Example::StaticMethod()
newobj instance void Example::.ctor()
callvirt instance void Example::InstanceMethod()
callvirt instance class [mscorlib]System.Action Example::get_Prop()
callvirt instance void [mscorlib]System.Action::Invoke()
Notice those call and callvirt instructions: these are method calls.
To actually execute the called methods, their IL code needs to be compiled into machine code (CPU instructions). This occurs in the virtual machine called .NET Runtime. There are several of them like .NET Framework, .NET Core, Mono etc.
A .NET Runtime contains a JIT (just-in-time) compiler. It converts the IL code to the actually executable code during the execution of your program.
When the .NET Runtime first encounters the IL code "call method StaticMethod from class Example", it first looks in the internal cache of already compiled methods. When there are no matches (which means this is the first call of that method), the Runtime asks the JIT compiler to create such a compiled-and-ready-to-run method using the IL code. The IL code is converted into a sequence of CPU operations and stored in the process' memory. A pointer to that compiled code is stored in the cache for future reuse.
This all will happen behind the call or callvirt IL instructions (again, simplified).
Once this happened, the Runtime is ready to execute the method. The CPU gets the compiled code's first operation address as the next operation to execute and goes on until the code returns. Then, the Runtime takes over again and proceeds with next IL instructions.
The DynamicInvoke method of the delegates does the same thing: it instructs the Runtime to call a method (after some additional arguments checks etc). The "dead end" you mention RuntimeMethodHandle.InvokeMethod is an intrinsic call to the Runtime directly. The parameters of this method are:
object target - the object on which the delegate invokes the instance method (this parameter).
object[] arguments - the arguments to pass to the method.
Signature sig - the actual method to call, Signature is an internal class that provides the connection between the managed IL code and native executable code.
bool constructor - true if this is a constructor call.
So in summary, methods are not represented as objects in C# (while you of course can have a delegate instance that is an object, but it doesn't represent the executable method, it rather provides an invokable reference to it).
Methods are called by the Runtime, the JIT compiler makes the methods executable.
You cannot define a global "function" outside of classes in C#. You could get a direct native pointer to the compiled (jitted) method code and probably even call it manually by directly manipulating own process' memory. But why?
You clearly misunderstand main differences between script languages, C/C++ and C#.
I guess the main difficulty is that there is no such thing as a function in C#. At all.
C#7 introduced the new feature "a local function", but that is not what a function in JS is.
All pieces of code are methods.
That name is intentionally different from function or a procedure to emphasize the fact that all executable code in C# belongs to a class.
Anonymous methods and lambdas are just a syntax sugar.
A compiler will generate a real method in the same (or a nested) class, where the method with anonymous method declaration belongs to.
This simple article explains it. You can take the examples, compile them and check the generated IL code yourself.
So all the methods (anonymous or not) do belong to a class. It's impossible to answer your updated question, besides saying It does not store a reference to a function, as there is no such thing in C#.
How does one store a reference to a method?
Depending on what you mean by reference, it can be either
An instance of MethodInfo class, used to reference reflection information for a method,
RuntimeMethodHandle (obtainable via RuntimeMethodInfo.MethodHandle) stores a real memory pointer to a JITed method code
A delegate, that is very different from just a memory pointer, but logically could be used to "pass a method reference to another method" .
I believe you are looking for the MethodInfo option, it has a MethodInfo.Invoke method which is very much alike Function..apply function in JS. You have already seen in the Delegate source code how that class is used.
If by "reference" you mean the C-style function pointer, it is in RuntimeMethodHandle struct. You should never use it without solid understanding how a particular .Net platform implementation and a C# compiler work.
Hopefully it clarifies things a bit.
A delegate is simply a pointer(memory location to jump to) to a method with the specified parameters and return type. Any Method that matches the signature(Parameters and return type) is eligible to fulfill the role, irrespective of the defined object. Anonymous simply means the delegate is not named.
Most times the type is implied(if it is not you will get a compiler error):
C# is a strongly typed language. That means every expression (including delegates) MUST have a return type(including void) as well as strongly typed parameters(if any). Generics were created to permit explicit types to be used within general contexts, such as Lists.
To put it another way, delegates are the type-safe managed version of C++ callbacks.
Delegates are helpful in eliminating switch statements by allowing the code to jump to the proper handler without testing any conditions.
A delegate is similar to a Closure in Javascript terminology.
In your response to Amy, you are attempting to equate a loosely typed language like JS, and a strongly typed language C#. In C# it is not possible to pass an arbitrary(loosely-typed) function anywhere. Lambdas and delegates are the only way to guarantee type safety.
I would recommend trying F#, if you are looking to pass functions around.
EDIT:
If you are trying to mimic the behavior of Javascipt, I would try looking at using inheritance through Interfaces. I can mimic multiple inheritance, and be type safe at the same time. But, be aware that it cannot fully supplant Javascript's dependency injection model.
As you probably found out C# doesn't have the concept of a function as in your JavaScript example.
C# is a statically typed language and the only way you can use function pointers is by using the built in types (Func,Action) or custom delegates.(I'm talking about safe,strongly typed pointers)
Javascript is a dynamic language that's why you can do what you describe
If you are willing to lose type safety, you can use the "dynamic" features of C# or refection to achieve what you want like in the following examples (Don't do this,use Func/Action)
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
namespace ConsoleApp1
{
class Program
{
private static Dictionary<string, Func<int, int, int>> FuncOps = new Dictionary<string, Func<int, int, int>>
{
{"add", (a, b) => a + b},
{"subtract", (a, b) => a - b}
};
//There are no anonymous delegates
//private static Dictionary<string, delegate> DelecateOps = new Dictionary<string, delegate>
//{
// {"add", delegate {} }
//};
private static Dictionary<string, dynamic> DynamicOps = new Dictionary<string, dynamic>
{
{"add", new Func<int, int, int>((a, b) => a + b)},
{"subtract", new Func<int, int, int>((a, b) => a - b)},
{"inverse", new Func<int, int>((a) => -a )} //Can't do this with Func
};
private static Dictionary<string, MethodInfo> ReflectionOps = new Dictionary<string, MethodInfo>
{
{"abs", typeof(Math).GetMethods().Single(m => m.Name == "Abs" && m.ReturnParameter.ParameterType == typeof(int))}
};
static void Main(string[] args)
{
Console.WriteLine(FuncOps["add"](3, 2));//5
Console.WriteLine(FuncOps["subtract"](3, 2));//1
Console.WriteLine(DynamicOps["add"](3, 2));//5
Console.WriteLine(DynamicOps["subtract"](3, 2));//1
Console.WriteLine(DynamicOps["inverse"](3));//-3
Console.WriteLine(ReflectionOps["abs"].Invoke(null, new object[] { -1 }));//1
Console.ReadLine();
}
}
}
one more example that you shouldn't use
delegate object CustomFunc(params object[] paramaters);
private static Dictionary<string, CustomFunc> CustomParamsOps = new Dictionary<string, CustomFunc>
{
{"add", parameters => (int) parameters[0] + (int) parameters[1]},
{"subtract", parameters => (int) parameters[0] - (int) parameters[1]},
{"inverse", parameters => -((int) parameters[0])}
};
Console.WriteLine(CustomParamsOps["add"](3, 2)); //5
Console.WriteLine(CustomParamsOps["subtract"](3, 2)); //1
Console.WriteLine(CustomParamsOps["inverse"](3)); //-3
I will provide a really short and simplified answer compared to the others. Everything in C# (classes, variables, properties, structs, etc) has a backed with tons of things your programs can hook into. This network of backend stuff slightly lowers the speed of C# when compared to "deeper" languages like C++, but also gives programmers a lot more tools to work with and makes the language easier to use. In this backend is included things like "garbage collection," which is a feature that automatically deletes objects from memory when there are no variables left that reference them. Speaking of reference, the whole system of passing objects by reference, which is default in C#, is also managed in the backend. In C#, Delegates are possible because of features in this backend that allow for something called "reflection."
From Wikipedia:
Reflection is the ability of a computer program to examine,
introspect, and modify its own structure and behavior at runtime.
So when C# compiles and it finds a Delegate, it is just going to make a function, and then store a reflective reference to that function in the variable, allowing you to pass it around and do all sorts of cool stuff with it. You aren't actually storing the function itself in the variable though, you are storing a reference, which is kinda like an address that points you to where the function is stored in RAM.
I'm playing around with the C# reflection API. I can easily load Type information of classes, methods etc. in an assembly, however, now I wonder how can I load and read the code inside a method?
Basic Answer:
You can't with the reflection API (System.Reflection).
The reason is that the reflection api is designed to work on Metadata (Type of Classes, Name and Signature of Methods, ...) but not on the data level (which would be the IL-stream itself).
Extended Answer:
You can emit (but not read) IL with System.Reflection.Emit (e.g. ILGenerator Class).
Through MethodInfo.GetMethodBody() you can get the binary IL-stream for the implementation of a method. But thats usually completely useless by itself.
There are external libraries (like Cecil) that you can use to read/modify/add/delete code inside a method.
That depends on what you mean by "read the code." There are 4 forms of the code.
Code Type
Can get with Reflection
The source code, i.e. the original C# or VB.NET
No
The symbolic IL code
No
The JITed assembly code
No
The IL bytes, i.e. the actual bytes that IL is compiled to
Yes
Take a look at MethodBase.GetMethodBody() for the last one. You can get the IL bytes, the local variables, exception frames etc.
You sort of can. The relevant function is MethodBase.GetMethodBody.
It's not exactly the most useful API. You can get some basic information about what's inside the method, and you can obtain the IL as a byte array. That's about it.
There's a slightly better API in the Mono.Cecil library, which exposes a MethodDefinition class with its own MethodBody implementation which contains actual Instructions, so you don't have to interpret the raw byte code. Still, if you're looking to get C# code out of it à la Reflector, you're going to be sorely disappointed. Also, Cecil isn't very well documented.
If you still want to try, then good luck.
If you don't need to do this real-time, have a look at Reflector. You can disassemble any .NET assembly (including the MS core DLLs) and see the code in your language of choice. This can be very educational.
Update Has anyone tried using Reflector on Reflector to figure out how this is done?
I'd like to provide an example of how one could explore the code inside a method. As others have explained, this can't be done easily using the native .NET Reflection API. However, using the Mono.Reflection API, you can disassemble the code programmatically using the GetInstructions() method and inspect it at runtime.
For example, the following code inspects a method and computes the number of calls inside it. As a use case for such a code, say I am a teacher (which I am) and instruct my fellow students to program a given method without using any other method, then using this code in unit tests I can verify that the given constrain is respected.
public static class MethodInfoUtil
{
public static int NbOfInnerCalls(this MethodInfo mi)
{
return mi.GetInstructions().Count(
instruction => instruction.OpCode.FlowControl == FlowControl.Call);
}
}
Example Console program:
class Program
{
static int Add(int a, int b) => a + b;
static int Doubling(int a) => Add(a, a);
static int Quadrupling(int a) => Add(Add(a, a), Add(a, a));
static void Main(string[] args)
{
Console.WriteLine("Inner method calls");
Console.WriteLine(" Add: {0}", ((Func<int, int, int>)Add).Method.NbOfInnerCalls());
Console.WriteLine(" Doubling: {0}", ((Func<int, int>)Doubling).Method.NbOfInnerCalls());
Console.WriteLine("Quadrupling: {0}", ((Func<int, int>)Quadrupling).Method.NbOfInnerCalls());
}
}
// Output:
// Inner method calls
// Add: 0
// Doubling: 1
// Quadrupling: 3
No
This is a feature slated for the next version of C#. You can use the CodeDom to get more info than reflection, but you cannot interrogate the parse tree yet.
Well there is always mono, in mono the compiler is a service, and you could get the parse trees at runtime.
The better question is why you want to?
Yes, there must be a way to achieve this: The .NET Reflector tool does this, too. Can't tell you how it's done there, though.
I have a property declared as type dynamic
public dynamic Data {get;set;}
later in some method the type of data becomes System.Collections.Generic.List
so if use Data.AsQueryable() i get "System.Collections.Generic.List<Entity1 does not contain a definition for 'AsQueryable' " error.
The result has to be converted to Iqueryble and i am using the methods defined in Dynamic.Linq.
How should i proceed?
Currently, dynamic doesn't work well with extension methods.
7.6.5.2 Extension method invocations
...if the normal processing of the
invocation finds no applicable
methods, an attempt is made to process
the construct as an extension method
invocation. If expr or any of the args
has compile-time type dynamic,
extension methods will not apply
As is mentioned in this question, the static context (applicable using directives) would have to be made available at run-time for every dynamic call to figure out which extension methods may apply, which is currently not implemented.
Have you tried calling the extension method as a 'normal' static method instead? E.g. (please modify if you intended to call a different method): System.Linq.Queryable.AsQueryable(Data)
Please, help me to explain the following behavior:
dynamic d = 1;
ISet<dynamic> s = new HashSet<dynamic>();
s.Contains(d);
The code compiles with no errors/warnings, but at the last line I get the following exception:
Unhandled Exception: Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: 'System.Collections.Generic.ISet<object>' does not contain a definition for 'Contains'
at CallSite.Target(Closure , CallSite , ISet`1 , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1)
at FormulaToSimulation.Program.Main(String[] args) in
As far as I can tell, this is related to dynamic overload resolution, but the strange things are
(1) If the type of s is HashSet<dynamic>, no exception occurs.
(2) If I use a non-generic interface with a method accepting a dynamic argument, no exception occurs.
Thus, it looks like this problem is related particularly with generic interfaces, but I could not find out what exactly causes the problem.
Is it a bug in the compiler/typesystem, or legitimate behavior?
The answers you have received so far do not explain the behaviour you are seeing. The DLR should find the method ICollection<object>.Contains(object) and call it with the boxed integer as a parameter, even if the static type of the variable is ISet<dynamic> instead of ICollection<dynamic> (because the former derives from the latter).
Therefore, I believe this is a bug and I have reported it to Microsoft Connect. If it turns out that the behaviour is somehow desirable, they will post a comment to that effect there.
Why it compiles: the entire expression is evaluated as dynamic (hover your mouse over it inside your IDE to confirm), which means that it is a runtime check.
Why it bombs: My (completely wrong, see below) guess is that it is because you cannot implement a dynamic interface in such a manner. For example, the compiler does not allow you to create a class that implements ISet<dynamic>, IEnumerable<dynamic>, IList<dynamic>, etc. You get a compile-time error stating "cannot implement a dynamic interface". See Chris Burrows' blog post on this subject.
http://blogs.msdn.com/b/cburrows/archive/2009/02/04/c-dynamic-part-vii.aspx
However, since it's hitting the DLR anyway, you can make s completely dynamic.
dynamic s = new HashSet<dynamic>;
s.Contains(d);
Compiles and runs.
Edit: the second part of this answer is completely wrong. Well, it is correct in that you can't implement such an interface as ISet<dynamic>, but that's not why this blows up.
See Julian's answer below. You can get the following code to compile and run:
ICollection<dynamic> s = new HashSet<dynamic>();
s.Contains(d);
The Contains method is defined on ICollection<T>, not ISet<T>. The CLR doesn't allow an interface base method to be called from a derived interface. You usually doesn't see this with static resolution because the C# compiler is smart enough to emit a call to ICollection<T>.Contains, not the non-existing ISet<T>.Contains.
Edit: The DLR mimics the CLR behavior, that's why you get the exception. Your dynamic call is done on an ISet<T>, not an HashSet<T> the DLR will mimics the CLR: for an interface, only interfaces methods are searched for, not base interfaces (contrary to classes where this behavior is present).
For an in-depth explanation, see a previous response of mine to a similar question:
Strange behaviour when using dynamic types as method parameters
Note that the type dynamic doesn’t actually exist at run-time. Variables of that type are actually compiled into variables of type object, but the compiler turns all the method calls (and properties and everything) that involve such an object (either as the this object or as a parameter) into a call that is resolved dynamically at runtime (using System.Runtime.CompilerServices.CallSiteBinder and related magic).
So what happens in your case is that the compiler:
turns ISet<dynamic> into ISet<object>;
turns HashSet<dynamic> into HashSet<object>, which becomes the actual run-time type of the instance you’re storing in s.
Now if you try to invoke, say,
s.Contains(1);
this actually succeeds without a dynamic invocation: it really just calls ISet<object>.Contains(object) on the boxed integer 1.
But if you try to invoke
s.Contains(d);
where d is dynamic, then the compiler turns the statement into one that determines, at runtime, the correct overload of Contains to call based on the runtime type of d. Perhaps now you can see the problem:
The compiler emits code that definitely searches the type ISet<object>.
That code determines that the dynamic variable has type int at runtime and tries to find a method Contains(int).
ISet<object> does not contain a method Contains(int), hence the exception.
ISet interface does not have a method 'Contains', HashSet does however?
EDIT
What i meant to say was the binder resolves 'Contains' when given the HashSet concreate type, but doesnt find the inherited 'Contains' method in the interface...
I have inherited a small scripting language and I am attempting to port it to the DLR so that it is a little easier to manage. So far it has been fairly straight forward. I have run into a problem though attempting to dynamically call members of a variable. The current language runs on .NET and uses a parsing loop and reflection to do this, but I was hoping to get away from that. Here is an example of the script language:
string $system1RemoteUri;
string $dbconnection = $config.GetDBConnection ("somedb");
float $minBad = 0.998;
float $minGood = 0.2;
$systen1RemoteURI, $minBad, and $minGood are variables that will be set in the script, along with $dbconnection. However $dbconnection will get its value from a variable passed in called $config. The 4 variables need to be available to the caller, so they are passed into the lambda, initially as null. Here is the generated Lambda IL (debug view):
.Lambda #Lambda1<Delegate6$1>(
System.String& $$system1RemoteUri,
System.String& $$dbconnection,
System.Double& $$minBad,
System.Double& $$minGood
System.Object $$config) {
.Block() {
$$minBad = 0.998D;
$$minGood = 0.2D
}
//Some assignment similar to...
//.Dynamic Call GetDBConnection($config, "somedb");
}
What I am trying to figure out is how to use Expression.Dynamic to emit the $config.GetDBConnection("somedb"). From looking at examples in the Sympl libraries I believe the emitted IL should look like:
.Dynamic Call GetdbConnection($config, "somedb") but I cant figure out how to actually emit that from Expression.Dynamic.
It seems to want a CallSiteBinder which I cannot create correctly, and I do not understand what the order of parameters is to Expression.Dynamic, as it seems to only want the "member" being invoked, and not the base.
I do not know the runtime type of $config it is just some object which implements a function called GetDBConnection(string). This is not provided by an interface or base class.
Any help would be appreciated.
You can either turn this into an InvokeMemberBinder or turn "$config.GetDBConnection" into a GetMember and then do an Invoke on the result of that passing $someDb as the argument.
To implement your GetMemberBinder and InvokeMemberBinder you can use the DLR outer-layer DefaultBinder class. In the latest IronPython/IronRuby source code you can just create a new DefaultBinder instance out of thin air. Then in your FallbackGetMember / FallbackInvoke you can call defaultBinder.GetMember(...) and defaultBinder.Call (which should be renamed Invoke). That'll deal with most .NET types for you. Also all objects which implement IDynamicMetaObjectProvider will work with it as well. For other dynamic operations you can use the other methods on the default binder. And if you want to start customizing your overload resolution and binding rules it has lots of knobs you can turn.
Unfortunately the default binder doesn't have an InvokeMemberBinder implementation right now so you're probably better off w/ GetMember/Invoke.