We Keep Coding

Does Linq's IEnumerable.Select return a reference to the original IEnumerable?

I was trying to clone an List in my code, because I needed to output that List to some other code, but the original reference was going to be cleared later on. So I had the idea of using the Select extension method to create a new reference to an IEnumerable of the same elements, for example:
List<int> ogList = new List<int> {1, 2, 3};
IEnumerable<int> enumerable = ogList.Select(s => s);
Now after doing ogList.Clear(), I was surprised to see that my new enumerable was also empty.
So I started fiddling around in LINQPad, and saw that even if my Select returned different objects entirely, the behaviour was the same.
List<int> ogList = new List<int> {1, 2, 3};
IEnumerable<int> enumerable = ogList.Select(s => 5); // Doesn't return the original int
enumerable.Count().Dump(); // Count is 3
ogList.Clear();
enumerable.Count().Dump(); // Count is 0!
Note that in LINQPad, the Dump()s are equivalent to Console.WriteLine().
Now probably my need to clone the list in the first place was due to bad design, and even if I didn't want to rethink the design I could easily clone it properly. But this got me thinking about what the Select extension method actually does.
According to the documentation for Select:
This method is implemented by using deferred execution. The immediate return value is an object that stores all the information that is required to perform the action. The query represented by this method is not executed until the object is enumerated either by calling its GetEnumerator method directly or by using foreach in Visual C# or For Each in Visual Basic.
So then I tried adding this code before clearing:
foreach (int i in enumerable)
{
i.Dump();
}
The result was still the same.
Finally, I tried one last thing to figure out if the reference in my new enumerable was the same as the old one. Instead of clearing the original List, I did:
ogList.Add(4);
Then I printed out the contents of my enumerable (the "cloned" one), expecting to see '4' appended to the end of it. Instead, I got:
5
5
5
5 // Huh?
Now I have no choice but to admit that I have no idea how the Select extension method works behind the scenes. What's going on?

List/List<T> are for all intents and purposes fancy resizable arrays. They own and hold the data for value types such as your ints or references to the data for reference types in memory and they always know how many items they have.
IEnumerable/IEnumerable<T> are different beasts. They provide a different service/contract. An IEnumerable is fictional, it does not exist. It can create data out of thin air, with no physical backing. Their only promise is that they have a public method called GetEnumerator() that returns an IEnumerator/IEnumerator<T>. The promise that an IEnumerator makes is simple:
some item could be available or not at a time when you decide you need it. This is achieved through a simple method that the IEnumerator interface has: bool MoveNext() - which returns false when the enumeration is completed or true if there was in fact a new item that needed to be returned. You can read the data through a property that the IEnumerator interface has, conveniently called Current.
To get back to your observations/question: as far as the IEnumerable in your example is concerned, it does not even think about the data unless your code tells it to fetch some data.
When you are writing:
List<int> ogList = new List<int> {1, 2, 3};
IEnumerable<int> enumerable = ogList.Select(s => s);
You are saying: Listen here IEnumerable, I might come to you asking for some items at some point in the future. I'll tell you when I will need them, for now sit still and do nothing. With Select(s => s) you are conceptually defining an identity projection of int to int.
A very rough simplified, non-real-life implementation of the select you've written is:
IEnumerable<T> Select(this IEnumerable<int> source, Func<int,T> transformer) something like
{
foreach (var i in source) //create an enumerator for source and starts enumeration
{
yield return transformer(i); //yield here == return an item and wait for orders
}
}
(this explains why you got a 5 when expecting a for, your transform was s => 5)
For value types, such as the ints in your case: If you want to clone the list, clone the whole list or part of it for future enumeration by using the result of an enumeration materialized through a List. This way you create a list that is a clone of the original list, entirely detached from its original list:
IEnumerable<int> cloneOfEnumerable = ogList.Select(s => s).ToList();
Later edit: Of course ogList.Select(s => s) is equivalent to ogList. I'm leaving the projection here, as it was in the question.
What you are creating here is: a list from the result of an enumerable, further consumed through the IEnumerable<int> interface. Considering what I've said above about the nature of IList vs IEnumerable, I would prefer to write/read:
IList<int> cloneOfEnumerable = ogList.ToList();
CAUTION: Be careful with reference types. IList/List make no promise of keeping the objects "safe", they can mutate to null for all IList cares. Keyword if you ever need it: deep cloning.
CAUTION: Beware of infinite or non-rewindable IEnumerables

Provided answers explain why you are not obtaining a cloned list (due to deferred execution of some LINQ extension methods).
However, keep in mind that list.Select(e => e).ToList() will get a real clone only when dealing with value types such as int.
If you have a list of reference types you will receive a cloned list of references to existent objects. In this case you should consider one of the solutions provided here for deep-cloning or my favorite from here (which might be limited by object inner structure).

You have to be aware that an object that implements IEnumerable does not have to be a collection itself. It is an object that makes it possible to get an object that implements IEnumerator. Once you have the enumerator you can ask for the first element and for the next element until there are no more next elements.
Every LINQ function that returns an IEnumerable is not the sequence itself, it only enables you to ask for the enumerator. If you want a sequence, you'll have to use ToList.
There are several other LINQ functions that do not return an IEnumerable, but for instance a Dictionary, or only one element (FirstOrDefault(), Max(), Single(), Any(). These functions will get the enumerator from the IEnumerable and start enumerating until they have the result. Any will only have to check if you can start enumerating. Max will enumerate over all elements and remember the largest one. etc.
You'll have to be aware: as long as your LINQ statement is an IEnumerable of something, your source sequence is not accessed yet. If you change your source sequence before you start enumerating, the enumeration is over your changed source sequence.
If you don't want this, you'll have to do the enumeration before you change your source. Usually this will be ToList, but this can be any of the non-deferred function: Max(), Any(), FirstOrDefault(), etc.
List<TSource> sourceItems = ...
var myEnumerable = sourceItems
.Where(sourceItem => ...)
.GroupBy(sourceItem => ...)
.Select(group => ...);
// note: myEnumerable is an IEnumerable, it is not a sequence yet.
var list1 = sourceItems.ToList(); // Enumerate over the sequence
var first = sourceItems.FirstOrDefault(); // Enumerate and stop after the first
// now change the source, and to the same things again
sourceItems.Clear();
var list1 = sourceItems.ToList(); // returns empty list
var first = sourceItems.FirstOrDefault(); // return null: there is no first element
So every LINQ function that does not return IEnumerable, will start enumerating over sourceItems as the sequence is at the moment that you start enumerating. The IEnumerable is not the sequence itself.

This is an enumerable.
var enumerable = ogList.Select(s => s);
If you iterate through this enumerable, LINQ will in turn iterate over the original resultset. Each and every time. If you do anything to the original enumerable, the results will also be reflected in your LINQ calls.
If you need to freeze the data, store it in a list instead:
var enumerable = ogList.Select(s => s).ToList();
Now you've made a copy. Iterating over this list will not touch the original enumerable.

C#: IEnumerable<T>.Select() inefficient in some cases?

I recently learned that the objects created by .NET's LINQ implementation is inefficient for specific enumeration types.
Take a look at this code:
public class DummyCollection : ICollection<int>
{
public IEnumerator<int> GetEnumerator()
{
throw new Exception();
}
public int Count
{
get
{
return 10;
}
}
//some more interface methods
}
basically, instances of DummyCollection have a size of 10, but throws an exception if it is actually enumerated.
now here:
var d = new DummyCollection();
Console.WriteLine(d.Count());
A 10 is printed without error, but this piece of code:
var l = d.Select(a=> a);
Console.WriteLine(l.Count());
throws an exception, despite it being trivial to say that l's size is 10 as well (since Select offers 1-to-1 mapping). What this basically means is, that when checking the length of an Ienumerable, the input might be a Select-wrapped Collection, thus extending the computation time from an O(1) to a staggering O(n) (could be even worse, if the selection function is particularly cumbersome).
I know that you sacrifice efficiency when you ask for LINQ's generics, but this seems like such a simple problem to fix. I checked online and couldn't find anyone addressing this. Is there a way to bypass this shortcoming? Is anyone looking into this? Is anyone fixing this? Is this just an edge case that isn't that much of a problem? Any insight is appreciated.

You can see how Count() extension method is implemented here. Basically is something like this:
public static int Count<TSource>(this IEnumerable<TSource> source)
{
if (source == null) throw Error.ArgumentNull("source");
ICollection<TSource> collectionoft = source as ICollection<TSource>;
if (collectionoft != null) return collectionoft.Count;
ICollection collection = source as ICollection;
if (collection != null) return collection.Count;
int count = 0;
using (IEnumerator<TSource> e = source.GetEnumerator()) {
checked {
while (e.MoveNext()) count++;
}
}
return count;
}
As you can see the method check first is the source is of type ICollection<TSource> or ICollection, if that is the case then there is no need to iterate counting the elements, just return Count property.
In your first case Count property is called returning 10 and GetEnumerator() method is never called.
When you use Select() method you're wrapping the collection into another type that isn't an ICollection (in above link you can also see Select() implementation), therefore the iteration is necessary.
In your second case, when you call Count() your GetEnumerator() method is called and the exception is thrown.

IEnumerable<T> doesn't have a concept of Count. This exists in implementations, which (apart from the odd shortcut here and there) have no role in LINQ to Objects. If you project an implementation of IEnumerable<T> (such as ICollection<T>), with Select, the only real guarantee you have is that the output will be IEnumerable<T>... which has no Count.
LINQ should be thought of as dealing with sequences of items, one at a time, only with a concept of current and next item (or the end of sequence). Knowing about the number of items is a (potentially) costly operation that requires iteration of all the items being counted, other than in a few, optimized cases.
Given that LINQ relies on iteration in preference to indexes and counts means that an IEnumerable that errors when you try to iterate it is going to need some super weird special-casing to fly. To me, it wouldn't be a very useful use-case.

ArrayList Count vs Any

I am looking at some legacy code. The class uses an ArrayList to keep the items. The items are fetched from Database table and can be up to 6 million. The class exposes a method called 'ListCount' to get the count of the items in the Arraylist.
Class Settings
{
ArrayList settingsList ;
public Settings()
{
settingsList = GetSettings();//Get the settings from the DB. Can also return null
}
public int ListCount
{
get
{
if (settingsList == null )
return 0;
else
return settingsList.Count;
}
}
}
The ListCount is used to check if there are items in the list. I am wondering to introduce 'Any' method to the class.
public bool Any(Func<vpSettings, bool> predicate)
{
return settingsList !=null && settingsList.Cast<vpSettings>().Any(predicate);
}
The question is does the framework do some kind of optimization and maintains a count of the items or does it iterate over the Arraylist to get the count? Would it be advisable to add the 'Any' method as above.
Marc Gravel in the following question advises to use Any for IEnumerable
Which method performs better: .Any() vs .Count() > 0?

The .NET reference source says that ArrayList.Count returns a cached private variable.
For completeness, the source also lists the implementation of the Any() extension method here. Essentially the extension method does a null check and then tries to get the first element via the IEnumerable's enumerator.

The ArrayList is actually implementing IList, which should be faster than the .Any(). The reason though because it is implementing the Count Property not the Count Method. The Count Property should do a quick check then grab the proper property.
Which looks similar to:
ICollection<TSource> collection1 = source as ICollection<TSource>;
if (collection1 != null)
return collection1.Count;
ICollection collection2 = source as ICollection;
if (collection2 != null)
return collection2.Count;

Marc Gravel advises to use Any() over Count() (the extension method), but not necessarily over Count (the property).
The Count property is always going to be faster, because it's just looking up an int that's stored on the heap. Using linq requires a (relatively) expensive object allocation to create the IEnumerator, plus whatever overhead there is in MoveNext (which, if the list is not empty, will needlessly copy the value of the ArrayList's first member to the Current property before returning true).
Now this is all pretty trivial for performance, but the code to do it is more complex, so it should only be used if there is a compelling performance benefit. Since there's actually a trivial performance penalty, we should choose the simpler code. I would therefore implement Any() as return Count > 0;.
However, your example is implementing the parameterized overload of Any. In that case, your solution, delegating to the parameterized Any extension method seems best. There's no relationship between the parameterized Any extension method and the Count property.

ArrayList implements IList so it does have a Count property. Using that would be faster than Any(), if all you care is check the container (non-)emptiness.

How does OrderBy extension method orders collection

Suppose we have a class Foo which has an Int32 Field Bar and you want to sort the collection of the Foo objects by the Bar Value. One way is to implement IComparable's CompareTo() method, but it can also be done using Language Integrated Query (LINQ) like this
List<Foo> foos = new List<Foo>();
// assign some values here
var sortedFoos = foos.OrderBy(f => f.Bar);
now in sortedFoos we have foos collection which is sorted. But if you use System.Diagnostics.StopWatch object to measure the time it took OrderBy() to sort the collection is always 0 milliseconds. But whenever you print sortedFoos collection it's obviously sorted. How is that possible. It takes literary no time to sort collection but after method execution the collection is sorted ? Can somebody explain to me how that works ? And also one thing. Suppose after sorting foos collection I added another element to it. now whenever I print out the collection the the element I added should be in the end right ? WRONG ! The foos collection will be sorted like, the element I added was the part of foos collection even if I add that element to foos after sorting. I don't quit understand how any of that work so can anybody make it clear for me ?!

Almost all LINQ methods use lazy evaluation - they don't immediately do anything, but they set up the query to do the right thing when you ask for the data.
OrderBy follows this model too - although it's less lazy than methods like Where and Select. When you ask for the first result from the result of OrderBy, it will read all the data from the source, sort all of it, and then return the first element. Compare this to Select for example, where asking for the first element from the projection only asks for the first element from the source.
If you're interested in how LINQ to Objects works behind the scenes, you might want to read my Edulinq blog series - a complete reimplementation of LINQ to Objects, with a blog post describing each method's behaviour and implementation.
(In my own implementation of OrderBy, I actually only sort lazily - I use a quick sort and sort "just enough" to return the next element. This can make things like largeCollection.OrderBy(...).First() a lot quicker.)

LINQ believe in deferred execution. This means the expression will only be evaluated when you started iterating or accessing the result.

The OrderBy extension uses the deault IComparer for the type it is working on, unless an alternative is passed via the appropriate overload.
The sorting work is defered until the IOrderedEnumerable<T> returned by your statement is first accessed. If you place the Stopwatch around that first access, you'll see how long sorting takes.
This makes a lot of sense, since your statement could be formed from multiple calls that return IOrderedEnumerables. As the ordering calls are chained, they fluently extend the result, allowing the finally returned IOrderedEnumerable to perform the sorting in the most expedient way possible. This is probably achieved by chaining all the IComparer calls and sorting once. A greedy implementation would have to wastefully sort multiple times.
For instance, consider
class MadeUp
{
public int A;
public DateTime B;
public string C;
public Guid D;
}
var verySorted = madeUps.OrderBy(m => m.A)
.ThenBy(m => m.B)
.ThenByDescending(m => m.C)
.ThenBy(m => m.D);
If verySorted was evaluated greedily, then every property in the sequence would be evaluated and the sequence would be reordered 4 times. Because the linq implementation of IOrderedEnumerable deferes sorting until enumeration, it is able to optimise the process.
The IComparers for A, B, C and D can be combined into a composite delegate, something like this simplified representation,
int result
result = comparerA(A, B);
if (result == 0)
{
result = comparerB(A, B);
if (result == 0)
{
result = comparerC(A, B);
if (result == 0)
{
result = comparerD(A, B);
}
}
}
return result;
the composite delegate is then used to sort the sequence once.

You have to add the ToList() to get a new collection. If you do't the OrderBy will be called when you start iterating on sortedFoos.
List<Foo> foos = new List<Foo>();
// assign some values here
var sortedFoos = foos.OrderBy(f => f.Bar).ToList();

When should I use .Count() and .Count in the context of an IEnumerable<T>

I'm aware that .Count() is an extension method in LINQ, and that fundamentally it uses the .Count, so I'm wondering, when should I use Count() and when should I use .Count? Is .Count() predominately better saved for queryable collections that are yet to be executed, and therefore don't have an enumeration yet? Am I safer simply always using .Count() extension method, or vice versa for the property? Or is this solely conditional depending on the collection?
Any advice, or articles, are greatly appreciated.
Update 1
After decompiling the .Count() extension method in LINQ it appears to be using the .Count property if the IEnumerable<T> is an ICollection<T> or ICollection, which is what most answers have suggested. The only real overhead now that I can see is the additional null and type checks, which isn't huge I suppose, but could still make a small amount of difference if performance were of the utmost importance.
Here's the decompiled LINQ .Count() extension method in .NET 4.0.
public static int Count<TSource>(this IEnumerable<TSource> source)
{
if (source == null)
{
throw Error.ArgumentNull("source");
}
ICollection<TSource> collection = source as ICollection<TSource>;
if (collection != null)
{
return collection.Count;
}
ICollection collection2 = source as ICollection;
if (collection2 != null)
{
return collection2.Count;
}
int num = 0;
checked
{
using (IEnumerator<TSource> enumerator = source.GetEnumerator())
{
while (enumerator.MoveNext())
{
num++;
}
}
return num;
}
}

The extension method works on any IEnumerable<T> but it is costly because it counts the sequence by iterating it. There is an optimization if the sequence is ICollection<T> meaning that the length of the collection is known. Then the Count property is used but that is an implementation detail.
The best advice is to use the Count property if available for performance reasons.
Is .Count() predominately better saved for queryable collections that are yet to be executed, and therefore don't have an enumeration yet?
If your collection is IQueryable<T> and not IEnumerable<T> then the query provider may be able to return the count in some efficient maner. In that case you will not suffer a performance penalty but it depends on the query provider.
An IQueryable<T> will not have a Count property so there is no choice between using the extension method and the property. However, if you query provider does not provide an efficient way of computing Count() you might consider using .ToList() to pull the collection to the client side. It really depends on how you intend to use it.

Count retrieves the property from a List (already calculated). Count() is an aggregation, like Sum(), Average(), etc. What it does is to count the items in the enumerable (I believe it internally uses the Count property if the enumerable is a list).
This is an example of the concrete use of the Count() method, when it doesn't just use the Count property:
var list = new List {1,2,3,4,5,6,7,8,9,10};
var count = list.Where(x => x > 5).Count();
Also, Count() has an overload that will count the items matching the predicate:
var count = list.Count(x => x > 5);

You should use Count() when all you have is an interface that doesn't expose a Count or Length property, such as IEnumerabe<T>.
If you're dealing with a collection or collection interface (such as List<T> or ICollection) then you can simply use the Count property, likewise if you have an array use the Length property.
The implementation of the Count() extension property will use the underlying collection's Count property if it is available. Otherwise the collection will be enumerated to calculate the count.

Agreed with comments of .Count if it is available (i.e. an object that implements ICollection<T> under the bonnet).
But they are wrong about .Count() being 'costly'. Enumerable.Count() will check if the object implements ICollection<T>.Count before it enumerates the elements and count them.
I.e. something like,
public int Enumerable.Count<TSource>(IEnumerable<TSource> source)
{
var collection = source as ICollection
if (collection != null)
{
return collection.Count;
}
}

I'm not sure it matters since Count() probably just reads the Count property. The performance difference is truly negligible. Use whichever you like. I use Count() when possible just to be consistent.

As others have said, if you have an ICollection<T>, use the Count property.
I would suggest that the IEnumerable.Count() method is really intended for use when the only thing you want to do with the elements of an enumeration is count them. The equivalent of SQL "SELECT COUNT(...".
If in addition you want to do something else with the elements of an enumeration, it makes more sense to generate a collection (usually a list using ToList()), then you can use the Count property and do whatever else you want to do.