We Keep Coding

Yield item in a list

In my application there is a List<MyItem> with a getter only:
public List<MyItem> myList
{
get
{
MyHost.GetItemFromID(_i1); //this may be a long operation
MyHost.GetItemFromID(_i2);
MyHost.GetItemFromID(_i3);
MyHost.GetItemFromID(_i4);
MyHost.GetItemFromID(_i5);
}
}
This list needs sometimes to be retrieved as whole and other times only certain item has to be accessed: i.e. myList[3]. Is there a way of not building the entire list as I only need the fourth item?

You could wrap the list with a class, say 'myListContainer' and overload its '[]' operator so that you could do something like this:
myListContainer[3]
which will invoke the call
MyHost.GetItemFromID(_i3);
and return the desired list item.
I'll add a full example if needed
EDIT
public class myListContainer
{
public MyItem this[int i]
{
get
{
return MyHost.GetItemFromID(i);
}
}
}
and add method to get the entire list.

You could return IEnumerable<MyItem> and use yield return:
public IEnumerable<MyItem> MyItems
{
get
{
yield return MyHost.GetItemFromID(_i1); //this may be a long operation
yield return MyHost.GetItemFromID(_i2);
yield return MyHost.GetItemFromID(_i3);
yield return MyHost.GetItemFromID(_i4);
yield return MyHost.GetItemFromID(_i5);
}
}
Then it is using deferred execution and you can write:
var fourItems = MyItems.Take(4).ToList();
Note that it might be a good idea to change the order of execution if the order doesn't matter and only the first call of GetItemFromID takes more time than the others:
yield return MyHost.GetItemFromID(_i2);
yield return MyHost.GetItemFromID(_i3);
yield return MyHost.GetItemFromID(_i4);
yield return MyHost.GetItemFromID(_i5);
yield return MyHost.GetItemFromID(_i1); //this may be a long operation
I think i have misuderstood the requirement. I've read "sometimes I only need the fourth item" as "only need four items"
So you could use my aproach but with ElementAt:
MyItem fourthItem = MyItems.ElementAt(3);
If you don't know if there's a fourth use ElementAtOrdefault.

I don't know what type your keys for _i1 etc are, but let's assume they are ints. Then you could do this:
public IEnumerable<MyItem> GetItems(params int[] keys)
{
return keys.Select(key => MyHost.GetItemFromID(key));
}
Which you could call like this:
var myItems = GetItems(_i2, _i3, _i5).ToList();
or
var myItems = GetItems(_i3).ToArray();
and so on.
Note: If you only ever want a list returned, you can do the conversion inside GetItems() itself:
public List<MyItem> GetItems(params int[] keys)
{
return keys.Select(key => MyHost.GetItemFromID(key)).ToList();
}
This approach requires a method rather than a property, though.

Note:
This assumes by '4th item' you mean an ordinal number, not the actual id itself.
Build a map in advance and return only items you wish to.
List<int> map = new List<int>
{
_i1,
_i2,
_i3,
_i4,
_i5,
};
public IEnumerable<MyItem> myList(params int[] indices)
{
if (!indices.Any())
return map.Select(MyHost.GetItemFromID);
return indices.Select(i => MyHost.GetItemFromID(map[i]));
}
// so you call
myList(); // for all items; decide on this API, may be separate to two methods?
myList(0); // or
myList(4); // or
myList(1, 3); // all loaded only on demand
Use a dictionary if you want more control over indexing. Ideally, for some reason this looks like you should be passing all IDs to DB in one go and SQL directly handle it, if that is the case.

C# yield return performance

How much space is reserved to the underlying collection behind a method using yield return syntax WHEN I PERFORM a ToList() on it? There's a chance it will reallocate and thus decrease performance if compared to the standard approach where i create a list with predefined capacity?
The two scenarios:
public IEnumerable<T> GetList1()
{
foreach( var item in collection )
yield return item.Property;
}
public IEnumerable<T> GetList2()
{
List<T> outputList = new List<T>( collection.Count() );
foreach( var item in collection )
outputList.Add( item.Property );
return outputList;
}

yield return does not create an array that has to be resized, like what List does; instead, it creates an IEnumerable with a state machine.
For instance, let's take this method:
public static IEnumerable<int> Foo()
{
Console.WriteLine("Returning 1");
yield return 1;
Console.WriteLine("Returning 2");
yield return 2;
Console.WriteLine("Returning 3");
yield return 3;
}
Now let's call it and assign that enumerable to a variable:
var elems = Foo();
None of the code in Foo has executed yet. Nothing will be printed on the console. But if we iterate over it, like this:
foreach(var elem in elems)
{
Console.WriteLine( "Got " + elem );
}
On the first iteration of the foreach loop, the Foo method will be executed until the first yield return. Then, on the second iteration, the method will "resume" from where it left off (right after the yield return 1), and execute until the next yield return. Same for all subsequent elements.
At the end of the loop, the console will look like this:
Returning 1
Got 1
Returning 2
Got 2
Returning 3
Got 3
This means you can write methods like this:
public static IEnumerable<int> GetAnswers()
{
while( true )
{
yield return 42;
}
}
You can call the GetAnswers method, and every time you request an element, it'll give you 42; the sequence never ends. You couldn't do this with a List, because lists have to have a finite size.

How much space is reserved to the underlying collection behind a method using yield return syntax?
There's no underlying collection.
There's an object, but it isn't a collection. Just how much space it will take up depends on what it needs to keep track of.
There's a chance it will reallocate
No.
And thus decrease performance if compared to the standard approach where i create a list with predefined capacity?
It will almost certainly take up less memory than creating a list with a predefined capacity.
Let's try a manual example. Say we had the following code:
public static IEnumerable<int> CountToTen()
{
for(var i = 1; i != 11; ++i)
yield return i;
}
To foreach through this will iterate through the numbers 1 to 10 inclusive.
Now let's do this the way we would have to if yield did not exist. We'd do something like:
private class CountToTenEnumerator : IEnumerator<int>
{
private int _current;
public int Current
{
get
{
if(_current == 0)
throw new InvalidOperationException();
return _current;
}
}
object IEnumerator.Current
{
get { return Current; }
}
public bool MoveNext()
{
if(_current == 10)
return false;
_current++;
return true;
}
public void Reset()
{
throw new NotSupportedException();
// We *could* just set _current back, but the object produced by
// yield won't do that, so we'll match that.
}
public void Dispose()
{
}
}
private class CountToTenEnumerable : IEnumerable<int>
{
public IEnumerator<int> GetEnumerator()
{
return new CountToTenEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public static IEnumerable<int> CountToTen()
{
return new CountToTenEnumerable();
}
Now, for a variety of reasons this is quite different to the code you're likely to get from the version using yield, but the basic principle is the same. As you can see there are two allocations involved of objects (same number as if we had a collection and then did a foreach on that) and the storage of a single int. In practice we can expect yield to store a few more bytes than that, but not a lot.
Edit: yield actually does a trick where the first GetEnumerator() call on the same thread that obtained the object returns that same object, doing double service for both cases. Since this covers over 99% of use cases yield actually does one allocation rather than two.
Now let's look at:
public IEnumerable<T> GetList1()
{
foreach( var item in collection )
yield return item.Property;
}
While this would result in more memory used than just return collection, it won't result in a lot more; the only thing the enumerator produced really needs to keep track of is the enumerator produced by calling GetEnumerator() on collection and then wrapping that.
This is going to be massively less memory than that of the wasteful second approach you mention, and much faster to get going.
Edit:
You've changed your question to include "syntax WHEN I PERFORM a ToList() on it", which is worth considering.
Now, here we need to add a third possibility: Knowledge of the collection's size.
Here, there is the possibilty that using new List(capacity) will prevent allocations of the list being built. That can indeed be a considerable saving.
If the object that has ToList called on it implements ICollection<T> then ToList will end up first doing a single allocation of an internal array of T and then calling ICollection<T>.CopyTo().
This would mean that your GetList2 would result in a faster ToList() than your GetList1.
However, your GetList2 has already wasted time and memory doing what ToList() will do with the results of GetList1 anyway!
What it should have done here was just return new List<T>(collection); and be done with it.
If though we need to actually do something inside GetList1 or GetList2 (e.g. convert elements, filter elements, track averages, and so on) then GetList1 is going to be faster and lighter on memory. Much lighter if we never call ToList() on it, and slightly ligher if we do call ToList() because again, the faster and lighter ToList() is offset by GetList2 being slower and heavier in the first place by exactly the same amount.

ToList method in Linq

If I am not wrong, the ToList() method iterate on each element of provided collection and add them to new instance of List and return this instance.Suppose an example
//using linq
list = Students.Where(s => s.Name == "ABC").ToList();
//traditional way
foreach (var student in Students)
{
if (student.Name == "ABC")
list.Add(student);
}
I think the traditional way is faster, as it loops only once, where as of above of Linq iterates twice once for Where method and then for ToList() method.
The project I am working on now has extensive use of Lists all over and I see there is alot of such kind of use of ToList() and other Methods that can be made better like above if I take list variable as IEnumerable and remove .ToList() and use it further as IEnumerable.
Do these things make any impact on performance?

Do these things make any impact on performance?
That depends on your code. Most of the time, using LINQ does cause a small performance hit. In some cases, this hit can be significant for you, but you should avoid LINQ only when you know that it is too slow for you (i.e. if profiling your code showed that LINQ is reason why your code is slow).
But you're right that using ToList() too often can cause significant performance problems. You should call ToList() only when you have to. Be aware that there are also cases where adding ToList() can improve performance a lot (e.g. when the collection is loaded from database every time it's iterated).
Regarding the number of iterations: it depends on what exactly do you mean by “iterates twice”. If you count the number of times MoveNext() is called on some collection, then yes, using Where() this way leads to iterating twice. The sequence of operations goes like this (to simplify, I'm going to assume that all items match the condition):
Where() is called, no iteration for now, Where() returns a special enumerable.
ToList() is called, calling MoveNext() on the enumerable returned from Where().
Where() now calls MoveNext() on the original collection and gets the value.
Where() calls your predicate, which returns true.
MoveNext() called from ToList() returns, ToList() gets the value and adds it to the list.
…
What this means is that if all n items in the original collection match the condition, MoveNext() will be called 2n times, n times from Where() and n times from ToList().

var list = Students.Where(s=>s.Name == "ABC");
This will only create a query and not loop the elements until the query is used. By calling ToList() will first then execute the query and thus only loop your elements once.
List<Student> studentList = new List<Student>();
var list = Students.Where(s=>s.Name == "ABC");
foreach(Student s in list)
{
studentList.add(s);
}
this example will also only iterate once. Because its only used once. Keep in mind that list will iterate all students everytime its called.. Not only just those whose names are ABC. Since its a query.
And for the later discussion Ive made a testexample. Perhaps its not the very best implementation of IEnumable but it does what its supposed to do.
First we have our list
public class TestList<T> : IEnumerable<T>
{
private TestEnumerator<T> _Enumerator;
public TestList()
{
_Enumerator = new TestEnumerator<T>();
}
public IEnumerator<T> GetEnumerator()
{
return _Enumerator;
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
throw new NotImplementedException();
}
internal void Add(T p)
{
_Enumerator.Add(p);
}
}
And since we want to count how many times MoveNext is called we have to implement our custom enumerator aswel. Observe in MoveNext we have a counter that is static in our program.
public class TestEnumerator : IEnumerator
{
public Item FirstItem = null;
public Item CurrentItem = null;
public TestEnumerator()
{
}
public T Current
{
get { return CurrentItem.Value; }
}
public void Dispose()
{
}
object System.Collections.IEnumerator.Current
{
get { throw new NotImplementedException(); }
}
public bool MoveNext()
{
Program.Counter++;
if (CurrentItem == null)
{
CurrentItem = FirstItem;
return true;
}
if (CurrentItem != null && CurrentItem.NextItem != null)
{
CurrentItem = CurrentItem.NextItem;
return true;
}
return false;
}
public void Reset()
{
CurrentItem = null;
}
internal void Add(T p)
{
if (FirstItem == null)
{
FirstItem = new Item<T>(p);
return;
}
Item<T> lastItem = FirstItem;
while (lastItem.NextItem != null)
{
lastItem = lastItem.NextItem;
}
lastItem.NextItem = new Item<T>(p);
}
}
And then we have a custom item that just wraps our value
public class Item<T>
{
public Item(T item)
{
Value = item;
}
public T Value;
public Item<T> NextItem;
}
To use the actual code we create a "list" with 3 entries.
public static int Counter = 0;
static void Main(string[] args)
{
TestList<int> list = new TestList<int>();
list.Add(1);
list.Add(2);
list.Add(3);
var v = list.Where(c => c == 2).ToList(); //will use movenext 4 times
var v = list.Where(c => true).ToList(); //will also use movenext 4 times
List<int> tmpList = new List<int>(); //And the loop in OP question
foreach(var i in list)
{
tmpList.Add(i);
} //Also 4 times.
}
And conclusion? How does it hit performance?
The MoveNext is called n+1 times in this case. Regardless of how many items we have.
And also the WhereClause does not matter, he will still run MoveNext 4 times. Because we always run our query on our initial list.
The only performance hit we will take is the actual LINQ framework and its calls. The actual loops made will be the same.
And before anyone asks why its N+1 times and not N times. Its because he returns false the last time when he is out of elements. Making it the number of elements + end of list.

To answer this completely, it depends on the implementation. If you are talking about LINQ to SQL/EF, there will be only one iteration in this case when .ToList is called, which internally calls .GetEnumerator. The query expression is then parsed into TSQL and passed to the database. The resulting rows are then iterated over (once) and added to the list.
In the case of LINQ to Objects, there is only one pass through the data as well. The use of yield return in the where clause sets up a state machine internally which keeps track of where the process is in the iteration. Where does NOT do a full iteration creating a temporary list and then passing those results to the rest of the query. It just determines if an item meets a criteria and only passes on those that match.

First of all, Why are you even asking me? Measure for yourself and see.
That said, Where, Select, OrderBy and the other LINQ IEnumerable extension methods, in general, are implemented as lazy as possible (the yield keyword is used often). That means that they do not work on the data unless they have to. From your example:
var list = Students.Where(s => s.Name == "ABC");
won't execute anything. This will return momentarily even if Students is a list of 10 million objects. The predicate won't be called at all until the result is actually requested somewhere, and that is practically what ToList() does: It says "Yes, the results - all of them - are required immediately".
There is however, some initial overhead in calling of the LINQ methods, so the traditional way will, in general, be faster, but composability and the ease-of-use of the LINQ methods, IMHO, more than compensate for that.
If you like to take a look at how these methods are implemented, they are available for reference from Microsoft Reference Sources.

Manually increment an enumerator inside foreach loop

I have a nested while loop inside a foreach loop where I would like to advance the enumerator indefinitately while a certain condition is met. To do this I try casting the enumerator to IEnumerator< T > (which it must be if it is in a foreach loop) then calling MoveNext() on the casted object but it gives me an error saying I cannot convert it.
Cannot convert type 'System.DateTime' to System.Collections.Generic.IEnumerator via a reference conversion, boxing conversion, unboxing conversion, wrapping conversion, or null type conversion.
foreach (DateTime time in times)
{
while (condition)
{
// perform action
// move to next item
(time as IEnumerator<DateTime>).MoveNext(); // will not let me do this
}
// code to execute after while condition is met
}
What is the best way to manually increment the IEnumerator inside of the foreach loop?
EDIT:
Edited to show there is code after the while loop that I would like executed once the condition is met which is why I wanted to manually increment inside the while then break out of it as opposed to continue which would put me back at the top. If this isn't possible I believe the best thing is to redesign how I am doing it.

Many of the other answers recommend using continue, which may very well help you do what you need to do. However, in the interests of showing manually moving the enumerator, first you must have the enumerator, and that means writing your loop as a while.
using (var enumerator = times.GetEnumerator())
{
DateTime time;
while (enumerator.MoveNext())
{
time = enumerator.Current;
// pre-condition code
while (condition)
{
if (enumerator.MoveNext())
{
time = enumerator.Current;
// condition code
}
else
{
condition = false;
}
}
// post-condition code
}
}
From your comments:
How can the foreach loop advance it if it doesn't implement the IEnumerator interface?
In your loop, time is a DateTime. It is not the object that needs to implement an interface or pattern to work in the loop. times is a sequence of DateTime values, it is the one that must implement the enumerable pattern. This is generally fulfilled by implementing the IEnumerable<T> and IEnumerable interfaces, which simply require T GetEnumerator() and object GetEnumerator() methods. The methods return an object implementing IEnumerator<T> and IEnumerator, which define a bool MoveNext() method and a T or object Current property. But time cannot be cast to IEnumerator, because it is no such thing, and neither is the times sequence.

You cannot modify the enumerator from inside the for loop. The language does not permit this. You need to use the continue statement in order to advance to the next iteration of a loop.
However, I'm not convinced that your loop even needs a continue. Read on.
In the context of your code you would need to convert the while to an if in order to make the continue refer to the foreach block.
foreach (DateTime time in times)
{
if (condition)
{
// perform action
continue;
}
// code to execute if condition is not met
}
But written like this it is clear that the following equivalent variant is simpler still
foreach (DateTime time in times)
{
if (condition)
{
// perform action
}
else
{
// code to execute if condition is not met
}
}
This is equivalent to your pseudo-code because the part marked code to execute after while condition is met is executed for each item for which condition is false.
My assumption in all of this is that condition is evaluated for each item in the list.

Perhaps you can use continue?

You would use the continue statement:
continue;

This is just a guess, but it sounds like what you're trying to do is take a list of datetimes and move past all of them which meet a certain criteria, then perform an action on the rest of the list. If that's what you're trying to do, you probably want something like SkipWhile() from System.Linq. For example, the following code takes a series of datetimes and skips past all of them which are before the cutoff date; then it prints out the remaining datetimes:
var times = new List<DateTime>()
{
DateTime.Now.AddDays(1), DateTime.Now.AddDays(2), DateTime.Now.AddDays(3), DateTime.Now.AddDays(4)
};
var cutoff = DateTime.Now.AddDays(2);
var timesAfterCutoff = times.SkipWhile(datetime => datetime.CompareTo(cutoff) < 1)
.Select(datetime => datetime);
foreach (var dateTime in timesAfterCutoff)
{
Console.WriteLine(dateTime);
}
Console.ReadLine();
Is that the sort of thing you're trying to do?

I definitely do not condone what I am about to suggest, but you can create a wrapper around the original IEnumerable to transform it into something that returns items which can be used to navigate the underlying the enumerator. The end result might look like the following.
public static void Main(string[] args)
{
IEnumerable<DateTime> times = GetTimes();
foreach (var step in times.StepWise())
{
while (condition)
{
step.MoveNext();
}
Console.WriteLine(step.Current);
}
}
Then we need to create our StepWise extension method.
public static class EnumerableExtension
{
public static IEnumerable<Step<T>> StepWise<T>(this IEnumerable<T> instance)
{
using (IEnumerator<T> enumerator = instance.GetEnumerator())
{
while (enumerator.MoveNext())
{
yield return new Step<T>(enumerator);
}
}
}
public struct Step<T>
{
private IEnumerator<T> enumerator;
public Step(IEnumerator<T> enumerator)
{
this.enumerator = enumerator;
}
public bool MoveNext()
{
return enumerator.MoveNext();
}
public T Current
{
get { return enumerator.Current; }
}
}
}

You could use a func as your iterator and keep the state that you are changing in that delegate to be evaluated each iteration.
public static IEnumerable<T> FunkyIEnumerable<T>(this Func<Tuple<bool, T>> nextOrNot)
{
while(true)
{
var result = nextOrNot();
if(result.Item1)
yield return result.Item2;
else
break;
}
yield break;
}
Func<Tuple<bool, int>> nextNumber = () =>
Tuple.Create(SomeRemoteService.CanIContinueToSendNumbers(), 1);
foreach(var justGonnaBeOne in nextNumber.FunkyIEnumerable())
Console.Writeline(justGonnaBeOne.ToString());

One alternative not yet mentioned is to have an enumerator return a wrapper object which allows access to itself in addition to the data element being enumerated. For sample:
struct ControllableEnumeratorItem<T>
{
private ControllableEnumerator parent;
public T Value {get {return parent.Value;}}
public bool MoveNext() {return parent.MoveNext();}
public ControllableEnumeratorItem(ControllableEnumerator newParent)
{parent = newParent;}
}
This approach could also be used by data structures that want to allow collections to be modified in controlled fashion during enumeration (e.g. by including "DeleteCurrentItem", "AddBeforeCurrentItem", and "AddAfterCurrentItem" methods).

Extensions for IEnumerable generic

I've got two extensions for IEnumerable:
public static class IEnumerableGenericExtensions
{
public static IEnumerable<IEnumerable<T>> InSetsOf<T>(this IEnumerable<T> source, int max)
{
List<T> toReturn = new List<T>(max);
foreach (var item in source)
{
toReturn.Add(item);
if (toReturn.Count == max)
{
yield return toReturn;
toReturn = new List<T>(max);
}
}
if (toReturn.Any())
{
yield return toReturn;
}
}
public static int IndexOf<T>(this IEnumerable<T> source, Predicate<T> searchPredicate)
{
int i = 0;
foreach (var item in source)
if (searchPredicate(item))
return i;
else
i++;
return -1;
}
}
Then I write this code:
Pages = history.InSetsOf<Message>(500);
var index = Pages.IndexOf(x => x == Pages.ElementAt(0));
where
public class History : IEnumerable
But as a result I've got not '0' as I've expected, but '-1'. I cant understand - why so?

When you write Pages.IndexOf(x => x == Pages.ElementAt(0));, you actually run InSetsOf many times, due to deferred execution (aka lazy). To expand:
Pages = history.InSetsOf<Message>(500) - this line doesn't run InSetsOf at all.
Pages.IndexOf - Iterates over Pages, so it starts executing InSetsOf once.
x == Pages.ElementAt(0) - this executes InSetsOf again, once for every element in the collection of Pages (or at least until searchPredicate return true, which doesn't happen here).
Each time you run InSetsOf you create a new list (specifically, a new first list, because you use ElementAt(0)). These are two different objects, so comparison of == between them fails.
An extremely simple fix would be to return a list, so Pages is not a deferred query, but a concrete collection:
Pages = history.InSetsOf<Message>(500).ToList();
Another option is to use SequenceEqual, though I'd recommend caching the first element anyway:
Pages = history.InSetsOf<Message>(500);
var firstPage = Pages.FirstOrDefault();
var index = Pages.IndexOf(x => x.SequenceEqual(firstPage));

Does your class T implement the IComparable? If not, your equality check might be flawed, as the framework does not know exactly when T= T. You would also get by just overriding equals on your class T I would guess.