We Keep Coding

C#: IEnumerable<T>.Select() inefficient in some cases?

I recently learned that the objects created by .NET's LINQ implementation is inefficient for specific enumeration types.
Take a look at this code:
public class DummyCollection : ICollection<int>
{
public IEnumerator<int> GetEnumerator()
{
throw new Exception();
}
public int Count
{
get
{
return 10;
}
}
//some more interface methods
}
basically, instances of DummyCollection have a size of 10, but throws an exception if it is actually enumerated.
now here:
var d = new DummyCollection();
Console.WriteLine(d.Count());
A 10 is printed without error, but this piece of code:
var l = d.Select(a=> a);
Console.WriteLine(l.Count());
throws an exception, despite it being trivial to say that l's size is 10 as well (since Select offers 1-to-1 mapping). What this basically means is, that when checking the length of an Ienumerable, the input might be a Select-wrapped Collection, thus extending the computation time from an O(1) to a staggering O(n) (could be even worse, if the selection function is particularly cumbersome).
I know that you sacrifice efficiency when you ask for LINQ's generics, but this seems like such a simple problem to fix. I checked online and couldn't find anyone addressing this. Is there a way to bypass this shortcoming? Is anyone looking into this? Is anyone fixing this? Is this just an edge case that isn't that much of a problem? Any insight is appreciated.

You can see how Count() extension method is implemented here. Basically is something like this:
public static int Count<TSource>(this IEnumerable<TSource> source)
{
if (source == null) throw Error.ArgumentNull("source");
ICollection<TSource> collectionoft = source as ICollection<TSource>;
if (collectionoft != null) return collectionoft.Count;
ICollection collection = source as ICollection;
if (collection != null) return collection.Count;
int count = 0;
using (IEnumerator<TSource> e = source.GetEnumerator()) {
checked {
while (e.MoveNext()) count++;
}
}
return count;
}
As you can see the method check first is the source is of type ICollection<TSource> or ICollection, if that is the case then there is no need to iterate counting the elements, just return Count property.
In your first case Count property is called returning 10 and GetEnumerator() method is never called.
When you use Select() method you're wrapping the collection into another type that isn't an ICollection (in above link you can also see Select() implementation), therefore the iteration is necessary.
In your second case, when you call Count() your GetEnumerator() method is called and the exception is thrown.

IEnumerable<T> doesn't have a concept of Count. This exists in implementations, which (apart from the odd shortcut here and there) have no role in LINQ to Objects. If you project an implementation of IEnumerable<T> (such as ICollection<T>), with Select, the only real guarantee you have is that the output will be IEnumerable<T>... which has no Count.
LINQ should be thought of as dealing with sequences of items, one at a time, only with a concept of current and next item (or the end of sequence). Knowing about the number of items is a (potentially) costly operation that requires iteration of all the items being counted, other than in a few, optimized cases.
Given that LINQ relies on iteration in preference to indexes and counts means that an IEnumerable that errors when you try to iterate it is going to need some super weird special-casing to fly. To me, it wouldn't be a very useful use-case.

How to implement ICollection<T> on an IEnumerable<T>

I would like to know how to program what Microsoft is suggesting from the MSDN guidelines for collections, which state the following:
AVOID using ICollection<T> or ICollection as a parameter just to access the
Count property. Instead, consider using IEnumerable<T> or IEnumerable and
dynamically checking whether the object implements ICollection<T> or ICollection.
In short, how do I implement ICollection on an IEnumerable? Microsoft has links all over that article, but no "And here is how you do this" link.
Here is my scenario. I have an MVC web app with a grid that will paginate and have sorting capability on some of the collections. For instance, on an Employee administration screen I display a list of employees in a grid.
Initially I returned the collection as IEnumerable. That was convenient when I didn't need to paginate. But now I'm faced with paginating and needing to extract the Count of employee records to do that. One workaround was to pass an employeeCount integer by ref to my getEmployeeRecords() method and assign the value within that method, but that's just messy.
Based on what I've seen here on StackOverflow, the general recommendation is to use IEnumerable instead of ICollection, or Collection, or IList, or List. So I'm not trying to open up a conversation about that topic. All I want to know is how to make an IEnumerable implement an ICollection, and extract the record count, so my code is more aligned with Microsoft's recommendation. A code sample or clear article demonstrating this would be helpful.
Thanks for your help!

One thing to note is that if you use LINQ's Count() method, it already does the type checking for you:
public static int Count<TSource>(this IEnumerable<TSource> source)
{
if (source == null) throw Error.ArgumentNull("source");
ICollection<TSource> collectionoft = source as ICollection<TSource>;
if (collectionoft != null) return collectionoft.Count;
ICollection collection = source as ICollection;
if (collection != null) return collection.Count;
int count = 0;
using (IEnumerator<TSource> e = source.GetEnumerator())
{
checked
{
while (e.MoveNext()) count++;
}
}
return count;
}

Initially I returned the collection as IEnumerable.
Well there's half your problem. Return types should be as explicit as possible. If you have a collection, make the return type that collection. (I forget where, but this is mentioned in the guidelines.)
Based on what I've seen here on StackOverflow, the general recommendation is to use IEnumerable instead of ICollection, or Collection, or IList, or List.
Some developers have an obsession with casting everything as IEnumerable. I have no idea why, as there is no guidance anywhere from Microsoft that says that is a good idea. (I do know that some think it somehow makes the return value immutable, but really anyone can cast it back to the base type and make changes to it. Or just use dynamic and never even notice you gave them an IEnumerable.)
That's the rule for return types and local variables. For parameters you should be as accepting as possible. In practice that means accepting either IEnumerable or IList depending on whether or not you need to access it by index.
AVOID using ICollection or ICollection as a parameter just to access the
Count property.
The reason for this is that if you need the Count, you probably need to access it by index as well. If not today, then tomorrow. So go ahead and use IList just in case.
(I'm not sure I agree, but it does make some sense.)
In short, how do I implement ICollection on an IEnumerable?
Short answer: the .Count() extension method. Make sure you import System.Linq.
Long answer:
int count = 0;
if (x is ICollection)
count = ((ICollection)x).Count;
else
foreach (var c in x)
count ++;

IEnumerable is an interface, and so is ICollection. It's the object's type that implements one or the other or both. You can check if an object implements ICollection with obj is ICollection.
Example:
public class MyCollection<T> : IEnumerable<T>, ICollection<T>
{
// ... Implemented methods
}
// ...
void Foo(IEnumerable<int> elements)
{
int count;
if (elements is ICollection<int>) {
count = ((ICollection<int>)elements).Count;
}
else {
// Use Linq to traverse the whole enumerable; less efficient, but correct
count = elements.Count();
}
}
// ...
MyCollection<int> myStuff;
Foo(myStuff);

Doesn't ICollection implement IEnumerable already? If you need a collection then you need a collection.

LINQ - IEnumerable.ToList() and Deferred Execution confusion

I have an IEnumerable variable named "query" which represents an entity framework query. If I do the following, will it enumerate the query more than once? My confusion is in "result" being an IEnumerable and not a List.
IEnumerable<T> result = query.ToList();
bool test = result.Any();
as apposed to this:
// Using List to only enumerate the query once
List<T> result = query.ToList();
bool test = result.Any();
Also, I know this is silly, but would it enumerate twice if I do the following, or would it somehow know that "query" was already enumerated even though the result of the enumeration is not being used?
List<T> result = query.ToList();
bool test = query.Any();
Thanks!

Once you're calling ToList or ToArray you will create an in-memory collection. From then on you're not dealing with the database anymore.
So even if you declare it as IEnumerable<T> it actually remains a List<T> after query.ToList().
Also, all related LINQ extension methods will check if the sequence can be casted to a collection type. You can see that for example in Enumerable.Count the Count property wil be used if possible:
public static int Count<TSource>(this IEnumerable<TSource> source) {
if (source == null) throw Error.ArgumentNull("source");
ICollection<TSource> collectionoft = source as ICollection<TSource>;
if (collectionoft != null) return collectionoft.Count;
ICollection collection = source as ICollection;
if (collection != null) return collection.Count;
int count = 0;
using (IEnumerator<TSource> e = source.GetEnumerator()) {
checked {
while (e.MoveNext()) count++;
}
}
return count;
}
According to your last question, if it makes a difference wether or not you use the list or again the query in this code snippet:
List<T> result = query.ToList();
bool test = query.Any();
Yes, in this case you are not using the list in memory but the query which will then ask the database again even if .Any is not as expensive as .ToList.

When you call ToList on your query it will be transformed into a list. The query will be evaluated right then, the items will be pulled out, and the list will be populated. From then on that List has no knowledge of the original query. No amount of manipulation of that list can in any way affect or evaluate that query, as it knows nothing about it.
It doesn't matter what you call that List, or what type of variable you stick it in, the list itself simply doesn't know anything about the IQueryable anymore. Iterating the variable holding the list multiple times will simply iterate that list multiple times.
In just the same way that the list doesn't know a thing about the query, the query doesn't know a thing about the list. It doesn't remember that it's items were put into a list and continue to return those items. (You can actually write query objects like this, in theory. It's called memoization, for the query to cache it's results when iterated and continue to provide objects from that cache when iterated later. EF doesn't memoize its queries by default, nor does it provide a tool for memoization by default, although 3rd party tools provide such extensions.) This means that the 3rd code snippet that you have will actually execute two separate database queries, not just one.

No the enumeration occour only in ToList().
http://msdn.microsoft.com/it-it/library/bb342261(v=vs.110).aspx

When should I use .Count() and .Count in the context of an IEnumerable<T>

I'm aware that .Count() is an extension method in LINQ, and that fundamentally it uses the .Count, so I'm wondering, when should I use Count() and when should I use .Count? Is .Count() predominately better saved for queryable collections that are yet to be executed, and therefore don't have an enumeration yet? Am I safer simply always using .Count() extension method, or vice versa for the property? Or is this solely conditional depending on the collection?
Any advice, or articles, are greatly appreciated.
Update 1
After decompiling the .Count() extension method in LINQ it appears to be using the .Count property if the IEnumerable<T> is an ICollection<T> or ICollection, which is what most answers have suggested. The only real overhead now that I can see is the additional null and type checks, which isn't huge I suppose, but could still make a small amount of difference if performance were of the utmost importance.
Here's the decompiled LINQ .Count() extension method in .NET 4.0.
public static int Count<TSource>(this IEnumerable<TSource> source)
{
if (source == null)
{
throw Error.ArgumentNull("source");
}
ICollection<TSource> collection = source as ICollection<TSource>;
if (collection != null)
{
return collection.Count;
}
ICollection collection2 = source as ICollection;
if (collection2 != null)
{
return collection2.Count;
}
int num = 0;
checked
{
using (IEnumerator<TSource> enumerator = source.GetEnumerator())
{
while (enumerator.MoveNext())
{
num++;
}
}
return num;
}
}

The extension method works on any IEnumerable<T> but it is costly because it counts the sequence by iterating it. There is an optimization if the sequence is ICollection<T> meaning that the length of the collection is known. Then the Count property is used but that is an implementation detail.
The best advice is to use the Count property if available for performance reasons.
Is .Count() predominately better saved for queryable collections that are yet to be executed, and therefore don't have an enumeration yet?
If your collection is IQueryable<T> and not IEnumerable<T> then the query provider may be able to return the count in some efficient maner. In that case you will not suffer a performance penalty but it depends on the query provider.
An IQueryable<T> will not have a Count property so there is no choice between using the extension method and the property. However, if you query provider does not provide an efficient way of computing Count() you might consider using .ToList() to pull the collection to the client side. It really depends on how you intend to use it.

Count retrieves the property from a List (already calculated). Count() is an aggregation, like Sum(), Average(), etc. What it does is to count the items in the enumerable (I believe it internally uses the Count property if the enumerable is a list).
This is an example of the concrete use of the Count() method, when it doesn't just use the Count property:
var list = new List {1,2,3,4,5,6,7,8,9,10};
var count = list.Where(x => x > 5).Count();
Also, Count() has an overload that will count the items matching the predicate:
var count = list.Count(x => x > 5);

You should use Count() when all you have is an interface that doesn't expose a Count or Length property, such as IEnumerabe<T>.
If you're dealing with a collection or collection interface (such as List<T> or ICollection) then you can simply use the Count property, likewise if you have an array use the Length property.
The implementation of the Count() extension property will use the underlying collection's Count property if it is available. Otherwise the collection will be enumerated to calculate the count.

Agreed with comments of .Count if it is available (i.e. an object that implements ICollection<T> under the bonnet).
But they are wrong about .Count() being 'costly'. Enumerable.Count() will check if the object implements ICollection<T>.Count before it enumerates the elements and count them.
I.e. something like,
public int Enumerable.Count<TSource>(IEnumerable<TSource> source)
{
var collection = source as ICollection
if (collection != null)
{
return collection.Count;
}
}

I'm not sure it matters since Count() probably just reads the Count property. The performance difference is truly negligible. Use whichever you like. I use Count() when possible just to be consistent.

As others have said, if you have an ICollection<T>, use the Count property.
I would suggest that the IEnumerable.Count() method is really intended for use when the only thing you want to do with the elements of an enumeration is count them. The equivalent of SQL "SELECT COUNT(...".
If in addition you want to do something else with the elements of an enumeration, it makes more sense to generate a collection (usually a list using ToList()), then you can use the Count property and do whatever else you want to do.

Selecting x number of elements from a System.Collections.Generic.List c#

What i need is a way to select the last 100 elements from a list, as list
public List<Model.PIP> GetPIPList()
{
if (Repository.PIPRepository.PIPList == null)
Repository.PIPRepository.Load();
return Repository.PIPRepository.PIPList.Take(100);
}
I get error like this
'System.Collections.Generic.IEnumerable' to 'System.Collections.Generic.List'. An explicit conversion exists (are you missing a cast?)

somelist.Reverse().Take(100).Reverse().ToList();
This would be much cheaper than ordering :) Also preserves the original ordering.

If your list is large, you'll get the best performance by rolling your own:
public static class ListExtensions
{
public static IEnumerable<T> LastItems<T>(this IList<T> list, int numberOfItems) //Can also handle arrays
{
for (int index = Math.Max(list.Count - numberOfItems, 0); index < list.Count; index++)
yield return list[index];
}
}
Why is this faster than using Skip()? If you have a list with 50,000 items, Skip() calls MoveNext() on the enumerator 49,900 times before it starts returning items.
Why is it faster than using Reverse()? Because Reverse allocates a new array large enough to hold the list's elements, and copies them into the array. This is especially good to avoid if the array is large enough to go on the large object heap.

EDIT: I missed that you said you wanted the last 100 items, and weren't able to do that yet.
To get the last 100 items:
return Repository.PIPRepository.PIPList
.OrderByDescending(pip=>pip.??).Take(100)
.OrderBy(pip=>pip.??);
...and then change your method signature to return IEnumerable<Model.PIP>
?? signifies what ever property you would be sorting on.
Joel also gives a great solution, based on counting the number of items in the last, and skipping all but 100 of them. In many cases, that probably works better. I didn't want to post the same solution in my edit! :)

Try:
public List<Model.PIP> GetPIPList()
{
if (Repository.PIPRepository.PIPList == null)
Repository.PIPRepository.Load();
return Repository.PIPRepository.PIPList.Take(100).ToList();
}

The .Take() method returns and IEnumerable<T> rather than a List<T>. This is a good thing, and you should strongly consider altering your method and your work habits to use IEnumerable<T> rather than List<T> as much as is practical.
Aside from that, .Take(100) will also return the first 100 elements, rather than the last 100 elements. You want something like this:
public IEnumerable<Model.PIP> GetPIPs()
{
if (Repository.PIPRepository.PIPList == null)
Repository.PIPRepository.Load();
return Repository.PIPRepository.PIPList.Skip(Math.Max(0,Repository.PIPRepository.PIPList.Count - 100));
}
If you really need a list rather than an enumerable (hint: you probably don't), it's still better to build this method using an IEnumerable and use .ToList() at the place where you call this method.
At some point in the future you'll want to go back and update your Load() code to also use IEnumerable, as well as code later on in the process. The ultimate goal here is to get to the point where you are effectively streaming your objects to the browser, and only ever one have one of them loaded into memory on your web server at a time. IEnumerable allows for this. List does not.