We Keep Coding

C# compiler fails with misleading error when calling generic extension method with a dynamic argument

While reading Jon Skeet's book C# In Depth 4th Edition, I was doing some tests on dynamic binding limitations related to extension methods.
As Jon says in that chapter, dynamic binding is not supported for extension methods in .NET, so the compiler prevents us from passing a dynamic value as an argument by raising the appropriate error CS1973. So for the following code:
//Provided this extension method
public static int GetFoo(this List<int> target, int value) => value * 10;
//And the following code at the call site
dynamic d = 1;
var items = new List<int> { 1, 2, 3 };
var result = items.GetFoo(d); //Error CS1973 as expected because
//we passed a dynamic argument
The error message is straightforward:
List<int> has no applicable method named GetFoo but appears to have an extension method by that name. Extension methods cannot be
dynamically dispatched. Consider casting the dynamic arguments or
calling the extension method without the extension method syntax.
No surprise until now. But if we change slightly the extension method to make it generic in order to accept a List<T> instead of List<int>, the compiler raises now a different error CS1929.
//The generic version of 'GetFoo' seen above
public static int GetFooGeneric<T>(this List<T> target, int value) => value * 10;
//Unexpectedly, the following call raises error CS1929 (instead of CS1973)
var result = items.GetFooGeneric(d);
And the error message is, in my opinion, as misleading as senseless:
List<int> does not contain a definition for GetFooGeneric and the best
extension method overload Extensions.GetFooGeneric<T>(List<T>, int) requires a receiver of type List<T>.
Misleading because it hides the non-support of dynamic binding for extension methods, and it doesn't make sense to have an instance of type List<T> as the receiver to call the extension method on.
Does anyone know what's happening behind the scenes that makes the compiler raise this misleading error message?
PS: As a side note, if we supply the type argument for that same generic method, the compiler then raises the appropriate error CS1973 again as expected.
//By helping the compiler explicitly, it raises CS1973 appropriately
var result = items.GetFooGeneric<int>(d);

Ouch. That's really bad! I don't recall if that one was my fault or not, but you are definitely right that this is a terrible error message.
It possibly is a result of some heuristics we put into the analyzer to deal with situations where you are actually typing the code in the editor and we need to do type inference on an incomplete or wrong call to an extension method in order to get the IntelliSense right; perhaps that is interacting badly with the dynamic argument?  But I would need to actually look at the code to refresh my memory there.
If I have time later today I'll look at the Roslyn sources and see if I recognize this code path.
I know that is not much of an answer to your question, but I haven't debugged through that code path since at least 2012 so my recall of those design choices is not what it once was. :)

You write: GetValueGeneric<T>(this List<T> target, int value) Note that you write Value
Calling items.GetFooGeneric(d); will result in CS1929 because your function does not exist.
I tryied items.GetValueGeneric(d); and it gives CS1973

Why can't the compiler tell the better conversion target in this overload resolution case? (covariance)

Understanding the C# Language Specification on overload resolution is clearly hard, and now I am wondering why this simple case fails:
void Method(Func<string> f)
{
}
void Method(Func<object> f)
{
}
void Call()
{
Method(() => { throw new NotSupportedException(); });
}
This gives compile-time error CS0121, The call is ambiguous between the following methods or properties: followed by my two Method function members (overloads).
What I would have expected was that Func<string> was a better conversion target than Func<object>, and then the first overload should be used.
Since .NET 4 and C# 4 (2010), the generic delegate type Func<out TResult> has been covariant in TResult, and for that reason an implicit conversion exists from Func<string> to Func<object> while clearly no implicit conversion can exist from Func<object> to Func<string>. So it would make Func<string> the better conversion target, and the overload resolution should pick the first overload?
My question is simply: What part of the C# Spec am I missing here?
Addition: This works fine:
void Call()
{
Method(null); // OK!
}

My question is simply: What part of the C# Spec am I missing here?
Summary:
You have found a minor known bug in the implementation.
The bug will be preserved for backwards compatibility reasons.
The C# 3 specification contained an error regarding how the "null" case was to be handled; it was fixed in the C# 4 specification.
You can reproduce the buggy behavior with any lambda where the return type cannot be inferred. For example: Method(() => null);
Details:
The C# 5 specification says that the betterness rule is:
If the expression has a type then choose the better conversion from that type to the candidate parameter types.
If the expression does not have a type and is not a lambda, choose the conversion to the type that is better.
If the expression is a lambda then first consider which parameter type is better; if neither is better and the delegate types have identical parameter lists then consider the relationship between the inferred return type of the lambda and the return types of the delegates.
So the intended behaviour is: first the compiler should check to see if one parameter type is clearly better than the other, regardless of whether the argument has a type. If that doesn't resolve the situation and the argument is a lambda, then check to see which of the inferred return type converted to the parameters' delegate types' return type is better.
The bug in the implementation is the implementation doesn't do that. Rather, in the case where the argument is a lambda it skips the type betterness check entirely and goes straight to the inferred return type betterness check, which then fails because there is no inferred return type.
My intention was to fix this for Roslyn. However, when I went to implement this, we discovered that making the fix caused some real-world code to stop compiling. (I do not recall what the real-world code was and I no longer have access to the database that holds the compatibility issues.) We therefore decided to maintain the existing small bug.
I note that the bug was basically impossible before I added delegate variance in C# 4; in C# 3 it was impossible for two different delegate types to be more or less specific, so the only rule that could apply was the lambda rule. Since there was no test in C# 3 that would reveal the bug, it was easy to write. My bad, sorry.
I note also that when you start throwing expression tree types into the mix, the analysis gets even more complicated. Even though Func<string> is better than Func<object>, Expression<Func<string>> is not convertible to Expression<Func<object>>! It would be nice if the algorithm for betterness was agnostic with respect to whether the lambda was going to an expression tree or a delegate, but it is in some ways not. Those cases get complicated and I don't want to labour the point here.
This minor bug is an object lesson in the importance of implementing what the spec actually says and not what you think it says. Had I been more careful in C# 3 to ensure that the code matched the spec then the code would have failed on the "null" case and it would then have been clear earlier that the C# 3 spec was wrong. And the implementation does the lambda check before the type check, which was a time bomb waiting to go off when C# 4 rolled around and suddenly that became incorrect code. The type check should have been done first regardless.

Well, you are right. What causes problem here is the delegate you are passing as an argument. It has no explicit return type, you are just throwing an exception. Exception is basically an object but it is not considered as a return type of a method. Since there is no return call following the exception throw, compiler is not sure what overload it should use.
Just try this
void Call()
{
Method(() =>
{
throw new NotSupportedException();
return "";
});
}
No problem with choosing an overload now because of explicitly stated type of an object passed to a return call. It does not matter that the return call is unreachable due to the exception throw, but now the compiler knows what overload it should use.
EDIT:
As for the case with passing null, frenkly, I don't know the answer.

Dynamic binding vs type inference

In The C# Programming language Bill Wagner says:
Many people confuse dynamic binding with type inference. Type
inference is statically bound. The compiler determines the type at
compile time. For example:
var i = 5; //i is an int (Compiler performs type inference)
Console.WriteLine(i); //Static binding to Console.WriteLine(int)
The compiler infers that i is an integer. All binding on the variable
i uses static binding.
Now, given this information and my own made-up dynamic scenerio:
dynamic i = 5; //Compiler punts
Console.WriteLine(i);//This is now dynamically bound
We know type inference is statically bound. This implies that there is no way a dynamic variable can use type inference to determine a type. How does a dynamic type get resolved without using type inference?
Update
To try and clarify...at runtime we must somehow figure out what type i is right? Because I assign a literal 5 then the runtime can infer that i is an int. Isn't that type inference rather than dynamic binding?

What distinction is Bill making?
The distinction that Bill is making is that many people think that:
var x = Whatever();
x.Foo();
will work out at runtime what method Foo to call based on the type of object returned at runtime by Whatever. That's not true; that would be
dynamic x = Whatever();
x.Foo();
The var just means "work out the type at compile time and substitute it in", not "work it out at runtime".
So if I have
dynamic i = 5;
Console.WriteLine(i);
What happens?
The compiler generates code that is morally like this:
object i = (object)5;
DynamicCallSite callSite = new DynamicCallSite(typeof(Console), "WriteLine"));
callSite.Invoke(i);
It is a bit more complicated than that; the call site is cached, for one thing. But this gives you the flavour of it.
The invocation method asks i for its type via GetType and then starts up a special version of the C# compiler that can understand reflection objects. It does overload resolution on the members of Console named WriteLine, and determines which overload of Console.WriteLine would have been called had i been typed as int in the first place.
It then generates an expression tree representing that call, compiles the expression tree into a delegate, caches it in the call site, and invokes the delegate.
The second time you do this, the cached call site looks in its cache and sees that the last time that i was int, a particular delegate was invoked. So the second time it skips creating the call site and doing overload resolution, and just invokes the delegate.
For more information, see:
http://ericlippert.com/2012/10/22/a-method-group-of-one/
http://ericlippert.com/2012/11/05/dynamic-contagion-part-one/
http://ericlippert.com/2012/11/09/dynamic-contagion-part-two/
A historical perspective on the feature can be obtained from Chris and Sam's blogs:
http://blogs.msdn.com/b/cburrows/archive/tags/dynamic/
http://blogs.msdn.com/b/samng/archive/tags/dynamic/
They did a lot of the implementation; however some of these article reflect outdated design choices. We never did go with "The Phantom Method" algorithm, regrettably. (Not a great algorithm, but a great name!)

Enumerating return list from simple.data

I am attempting to process results from a select (Find.., All, etc) from simple.data, but I am getting errors:
var db = Database.Open();
var codes = db.Code.All();
// 'object' does not contain a definition for 'First'
var firstcode = codes.First();
// 'object' does not contain a definition for 'ToList'
List<Code> codeList = codes.ToList();
The type of codes is {System.Linq.Enumerable.WhereSelectListIterator<System.Collections.Generic.IDictionary<string,object>,Simple.Data.DynamicRecord>}.
What am I missing? Someone add a simple.data tag please.. :)

The main reason that LINQ methods don't work on the object returned from db.Code.All() is that at that point in the code, the C# compiler doesn't know that it's an IEnumerable, so it can't hook up the extension methods. Of course, the C# compiler doesn't know what the object is, because it's dynamic, so it passes over it and assumes that the First() method will be resolved at runtime.
I've tried to address this in more recent releases, and many methods are supported, including ToList, First, FirstOrDefault, Single, SingleOrDefault and some others. Still more are to come soon (within the 0.9.x releases).
The easiest way to bring the compiler back into full effect is to explicitly specify the type instead of using var. For example
IEnumerable<Code> codes = db.Codes.All();
will cause an "implicit cast" (quoted because it's not really, but it acts like one) from the SimpleQuery type to the IEnumerable type, at which point you can start using LINQ methods on it again.

Doh, simple answer. I was using the latest version from https://github.com/markrendle/Simple.Data/downloads but in fact it should installed from nuget http://nuget.org/List/Packages/Simple.Data.Core.. :(

It looks like you need a using System.Linq; declaration.
If the actual error message contains the word 'object' then it indicates that the type of codes returned from your call is an object, not a System.Linq.Enumerable.WhereSelectListIterator<System.Collections.Generic.IDictionary<string,object>,Simple.Data.DynamicRecord>} as you say, which would be the cause of the error.

Delegate as first param to an Extension Method

Ladies and Gents,
I recently tried this experiment:
static class TryParseExtensions
{
public delegate bool TryParseMethod<T>(string s, out T maybeValue);
public static T? OrNull<T>(this TryParseMethod<T> tryParser, string s) where T:struct
{
T result;
return tryParser(s, out result) ? (T?)result : null;
}
}
// compiler error "'int.TryParse(string, out int)' is a 'method', which is not valid in the given context"
var result = int.TryParse.OrNull("1"); // int.TryParse.OrNull<int>("1"); doesnt work either
// compiler error: type cannot be infered....why?
var result2 = TryParseExtensions.OrNull(int.TryParse, "2");
// works as expected
var result3 = TryParseExtensions.OrNull<int>(int.TryParse, "3");
var result4 = ((TryParseExtensions.TryParseMethod<int>)int.TryParse).OrNull("4");
I am wondering two things:
Why can the compiler not infer the "int" type parameter?
Do I understand correctly that extensions methods do not get discovered on Delegate types, as I guess they arent really of that type (but are a "Method") that only happen to match the delegates signature? As such a cast solves this. Would it be infeasable to enable scenario 1 to work (not this one specifically of course, but in general)? I guess from a language/compiler perspective and would it actually be useful, or am I just (attempting to) wildly abusing things here?
Looking forward to some insights. Thnx

You have a number of questions here. (In the future I would recommend that when you have multiple questions, split them up into multiple questions rather than one posting with several questions in it; you'll probably get better responses.)
Why can the compiler not infer the "int" type parameter in:
TryParseExtensions.OrNull(int.TryParse, "2");
Good question. Rather than answer that here, I refer you to my 2007 article which explains why this did not work in C# 3.0:
http://blogs.msdn.com/b/ericlippert/archive/2007/11/05/c-3-0-return-type-inference-does-not-work-on-member-groups.aspx
Summing up: fundamentally there is a chicken-and-egg problem here. We must do overload resolution on int.TryParse to determine which overload of TryParse is the intended one (or, if none of them work, what the error is.) Overload resolution always tries to infer from arguments. In this case though, it is precisely the type of the argument that we are attempting to infer.
We could come up with a new overload resolution algorithm that says "well, if there's only one method in the method group then pick that one even if we don't know what the arguments are", but that seems weak. It seems like a bad idea to special-case method groups that have only one method in them because that then penalizes you for adding new overloads; it can suddenly be a breaking change.
As you can see from the comments to that article, we got a lot of good feedback on it. The best feedback was got was basically "well, suppose type inference has already worked out the types of all the argument and it is the return type that we are attempting to infer; in that case you could do overload resolution". That analysis is correct, and changes to that effect went into C# 4. I talked about that a bit more here:
http://blogs.msdn.com/b/ericlippert/archive/2008/05/28/method-type-inference-changes-part-zero.aspx
Do I understand correctly that extensions methods do not get discovered on delegate types, as I guess they arent really of that type (but are a "Method") that only happen to match the delegates signature?
Your terminology is a bit off, but your idea is correct. We do not discover extension methods when the "receiver" is a method group. More generally, we do not discover extension methods when the receiver is something that lacks its own type, but rather takes on a type based on its context: method groups, lambdas, anonymous methods and the null literal all have this property. It would be really bizarre to say null.Whatever() and have that call an extension method on String, or even weirder, (x=>x+1).Whatever() and have that call an extension method on Func<int, int>.
The line of the spec which describes this behaviour is :
An implicit identity, reference or boxing conversion [must exist] from [the receiver expression] to the type of the first parameter [...].
Conversions on method groups are not identity, reference or boxing conversions; they are method group conversions.
Would it be infeasable to enable scenario 1 to work (not this one specifically of course, but in general)? I guess from a language/compiler perspective and would it actually be useful, or am I just (attempting to) wildly abusing things here?
It is not infeasible. We've got a pretty smart team here and there's no theoretical reason why it is impossible to do so. It just doesn't seem to us like a feature that adds more value to the language than the cost of the additional complexity.
There are times when it would be useful. For example, I'd like to be able to do this; suppose I have a static Func<A, R> Memoize<A, R>(this Func<A, R> f) {...}:
var fib = (n=>n<2?1:fib(n-1)+fib(n-2)).Memoize();
Instead of what you have to write today, which is:
Func<int, int> fib = null;
fib = n=>n<2?1:fib(n-1)+fib(n-2);
fib = fib.Memoize();
But frankly, the additional complexity the proposed feature adds to the language is not paid for by the small benefit in making the code above less verbose.

The reason for the first error:
int.TryParse is a method group, not an object instance of any type. Extension methods can only be called on object instances. That's the same reason why the following code is invalid:
var s = int.TryParse;
This is also the reason why the type can't be inferred in the second example: int.TryParse is a method group and not of type TryParseMethod<int>.
I suggest, you use approach three and shorten the name of that extension class. I don't think there is any better way to do it.

Note that your code works if you first declare :
TryParseExtensions.TryParseMethod<int> tryParser = int.TryParse;
and then use tryParser where you used int.TryParse.
The problem is that the compiler doesn't know which overload of int.Parse you're speaking about. So it cannot completely infer it : are you speaking about TryParse(String, Int32) or TryParse(String, NumberStyles, IFormatProvider, Int32) ? The compiler can't guess and won't arbitrarily decide for you (fortunately !).
But your delegate type makes clear which overload you're interested in. That's why assigning tryParser is not a problem. You're not speaking anymore of a "method group" but of a well identified method signature inside this group of methods called int.TryParse.