We Keep Coding

Is there a way to handle any type of collection, instead of solely relying on Array, List, etc?

This example is for a method called "WriteLines", which takes an array of strings and adds them to an asynchronous file writer. It works, but I am curious if there is an interesting way to support -any- collection of strings, rather than relying on the programmer to convert to an array.
I came up with something like:
public void AddLines(IEnumerable<string> lines)
{
// grab the queue
lock (_queue)
{
// loop through the collection and enqueue each line
for (int i = 0, count = lines.Count(); i < count; i++)
{
_queue.Enqueue(lines.ElementAt(i));
}
}
// notify the thread it has work to do.
_hasNewItems.Set();
}
This appears to work but I have no idea of any performance implications it has, or any logic implications either (What happens to the order? I assume this will allow even unordered collections to work, e.g. HashSet).
Is there a more accepted way to achieve this?

You've been passed an IEnumerable<string> - that means you can iterate over it. Heck, there's even a language feature specifically for it - foreach:
foreach (string line in lines)
{
_queue.Enqueue(line);
}
Unlike your existing approach, this will only iterate over the sequence once. Your current code will behave differently based on the underlying implementation - in some cases Count() and ElementAt are optimized, but in some cases they aren't. You can see this really easily if you use an iterator block and log:
public IEnumerable<string> GetItems()
{
Console.WriteLine("yielding a");
yield return "a";
Console.WriteLine("yielding b");
yield return "b";
Console.WriteLine("yielding c");
yield return "c";
}
Try calling AddLines(GetItems()) with your current implementation, and look at the console...

Adding this answer as well since you are using threads, use a ConcurrentQueue instead, like so:
// the provider method
// _queue = new BlockingCollection<string>()
public void AddLines(IEnumerable<string> lines)
{
foreach (var line in lines)
{
_queue.Add(line);
}
}
No locks required, and allows for multiple consumers and providers since we flag for each element added.
The consumer basically only has to do var workitem = _queue.Take();

Why prefer Yield over a simple return?

I am trying to understand use of Yield to enumerate the collection. I have written this basic code:
static void Main(string[] args)
{
Iterate iterate = new Iterate();
foreach (int i in iterate.EnumerateList())
{
Console.Write("{0}", i);
}
Console.ReadLine();
}
class Iterate
{
public IEnumerable<int> EnumerateList()
{
List<int> lstNumbers = new List<int>();
lstNumbers.Add(1);
lstNumbers.Add(2);
lstNumbers.Add(3);
lstNumbers.Add(4);
lstNumbers.Add(5);
foreach (int i in lstNumbers)
{
yield return i;
}
}
}
(1) What if I use simply return i instead of yield return i?
(2) What are the advantages of using Yield and when to prefer using it?
Edited **
In the above code, I think it is an overhead to use foreach two times. First in the main function and the second in the EnumerateList method.

Using yield return i makes this method an iterator. It will create an IEnumerable<int> sequence of values from your entire loop.
If you used return i, it would just return a single int value. In your case, this would cause a compiler error, as the return type of your method is IEnumerable<int>, not int.
In this specific example, I would personally just return lstNumbers instead of using the iterator. You could rewrite this without the list, though, as:
public IEnumerable<int> EnumerateList()
{
yield return 1;
yield return 2;
yield return 3;
yield return 4;
yield return 5;
}
Or even:
public IEnumerable<int> EnumerateList()
{
for (int i=1;i<=5;++i)
yield return i;
}
This is very handy when you're making a class which you want to act like a collection. Implementing IEnumerable<T> by hand for a custom type often requires making a custom class, etc. Prior to iterators, this required a lot of code, as shown in this old sample for implementing a custom collection in C#.

As Reed says using yield allows you to implement an iterator. The major advantage of an iterator is that it allows lazy evaluation. I.e. it doesn't have to materialize the entire result unless needed.
Consider Directory.GetFiles from the BCL. It returns string[]. I.e. it has to get all the files and put the names in an array before it returns. In contrast Directory.EnumerateFiles returns IEnumerable<string>. I.e. the caller is responsible for handling the result set. That means that the caller can opt out of enumerating the collection at any point.

The yield keyword will make the compiler turn the function into an enumerator object.
The yield return statement doesn't simply exit the function and return one value, it leaves the code in a state so that it can be resumed at that point when the next value is requested from the enumerator.

You can't return just i as i is an int and not an IEnumerable. It won't even compile. You could have returned lstNumbers as that implements the interface of IEnumerable.
I prefer yield return since the compiler will handle building the enumerable and not having to build the list in the first place. So for me if I have something that is already implementing the interface then I return that if I have to build it and not use it else where then I yield return.

Need help understanding C# yield in IEnumerable

i am reading C# 2010 Accelerated. i dont get what is yield
When GetEnumerator is called, the code
in the method that contains the yield
statement is not actually executed at
that point in time. Instead, the
compiler generates an enumerator
class, and that class contains the
yield block code
public IEnumerator<T> GetEnumerator() {
foreach( T item in items ) {
yield return item;
}
}
i also read from Some help understanding “yield”
yield is a lazy producer of data, only
producing another item after the first
has been retrieved, whereas returning
a list will return everything in one
go.
does this mean that each call to GetEnumerator will get 1 item from the collection? so 1st call i get 1st item, 2nd, i get the 2nd and so on ... ?

Best way to think of it is when you first request an item from an IEnumerator (for example in a foreach), it starts running trough the method, and when it hits a yield return it pauses execution and returns that item for you to use in your foreach. Then you request the next item, it resumes the code where it left and repeats the cycle until it encounters either yield break or the end of the method.
public IEnumerator<string> enumerateSomeStrings()
{
yield return "one";
yield return "two";
var array = new[] { "three", "four" }
foreach (var item in array)
yield return item;
yield return "five";
}

Take a look at the IEnumerator<T> interface; that may well to clarify what's happening. The compiler takes your code and turns it into a class that implements both IEnumerable<T> and IEnumerator<T>. The call to GetEnumerator() simply returns the class itself.
The implementation is basically a state machine, which, for each call to MoveNext(), executes the code up until the next yield return and then sets Current to the return value. The foreach loop uses this enumerator to walk through the enumerated items, calling MoveNext() before each iteration of the loop. The compiler is really doing some very cool things here, making yield return one of the most powerful constructs in the language. From the programmer's perspective, it's just an easy way to lazily return items upon request.

Yes thats right, heres the example from MSDN that illustrates how to use it
public class List
{
//using System.Collections;
public static IEnumerable Power(int number, int exponent)
{
int counter = 0;
int result = 1;
while (counter++ < exponent)
{
result = result * number;
yield return result;
}
}
static void Main()
{
// Display powers of 2 up to the exponent 8:
foreach (int i in Power(2, 8))
{
Console.Write("{0} ", i);
}
}
}
/*
Output:
2 4 8 16 32 64 128 256
*/

If I understand your question correct then your understanding is incorrect I'm affraid. The yield statements (yield return and yield break) is a very clever compiler trick. The code in you method is actually compiled into a class that implements IEnumerable. An instance of this class is what the method will return. Let's Call the instance 'ins' when calling ins.GetEnumerator() you get an IEnumerator that for each Call to MoveNext() produced the next element in the collection (the yield return is responsible for this part) when the sequence has no more elements (e.g. a yield break is encountered) MoveNext() returns false and further calls results in an exception. So it is not the Call to GetEnumerator that produced the (next) element but the Call to MoveNext

It looks like you understand it.
yield is used in your class's GetEnumerator as you describe so that you can write code like this:
foreach (MyObject myObject in myObjectCollection)
{
// Do something with myObject
}
By returning the first item from the 1st call the second from the 2nd and so on you can loop over all elements in the collection.
yield is defined in MyObjectCollection.

The Simple way to understand yield keyword is we do not need extra class to hold the result of iteration when return using
yield return keyword. Generally when we iterate through the collection and want to return the result, we use collection object
to hold the result. Let's look at example.
public static List Multiplication(int number, int times)
{
List<int> resultList = new List<int>();
int result = number;
for(int i=1;i<=times;i++)
{
result=number*i;
resultList.Add(result);
}
return resultList;
}
static void Main(string[] args)
{
foreach(int i in Multiplication(2,10))
{
Console.WriteLine(i);
}
Console.ReadKey();
}
In the above example, I want to return the result of multiplication of 2 ten times. So I Create a method Multiplication
which returns me the multiplication of 2 ten times and i store the result in the list and when my main method calls the
multiplication method, the control iterates through the loop ten times and store result result in the list. This is without
using yield return. Suppose if i want to do this using yield return it looks like
public static IEnumerable Multiplication(int number, int times)
{
int result = number;
for(int i=1;i<=times;i++)
{
result=number*i;
yield return result;
}
}
static void Main(string[] args)
{
foreach(int i in Multiplication(2,10))
{
Console.WriteLine(i);
}
Console.ReadKey();
}
Now there is slight changes in Multiplication method, return type is IEnumerable and there is no other list to hold the
result because to work with Yield return type must be IEnumerable or IEnumerator and since Yield provides stateful iteration
we do not need extra class to hold the result. So in the above example, when Multiplication method is called from Main
method, it calculates the result in for 1st iteration and return the result to main method and come backs to the loop and
calculate the result for 2nd iteration and returns the result to main method.In this way Yield returns result to calling
method one by one in each iteration.There is other Keyword break used in combination with Yield that causes the iteration
to stop. For example in the above example if i want to calculate multiplication for only half number of times(10/2=5) then
the method looks like this:
public static IEnumerable Multiplication(int number, int times)
{
int result = number;
for(int i=1;i<=times;i++)
{
result=number*i;
yield return result;
if (i == times / 2)
yield break;
}
}
This method now will result multiplication of 2, 5 times.Hope this will help you understand the concept of Yield. For more
information please visit http://msdn.microsoft.com/en-us/library/9k7k7cf0.aspx

Why do we need iterators in c#?

Can somebody provide a real life example regarding use of iterators. I tried searching google but was not satisfied with the answers.

You've probably heard of arrays and containers - objects that store a list of other objects.
But in order for an object to represent a list, it doesn't actually have to "store" the list. All it has to do is provide you with methods or properties that allow you to obtain the items of the list.
In the .NET framework, the interface IEnumerable is all an object has to support to be considered a "list" in that sense.
To simplify it a little (leaving out some historical baggage):
public interface IEnumerable<T>
{
IEnumerator<T> GetEnumerator();
}
So you can get an enumerator from it. That interface (again, simplifying slightly to remove distracting noise):
public interface IEnumerator<T>
{
bool MoveNext();
T Current { get; }
}
So to loop through a list, you'd do this:
var e = list.GetEnumerator();
while (e.MoveNext())
{
var item = e.Current;
// blah
}
This pattern is captured neatly by the foreach keyword:
foreach (var item in list)
// blah
But what about creating a new kind of list? Yes, we can just use List<T> and fill it up with items. But what if we want to discover the items "on the fly" as they are requested? There is an advantage to this, which is that the client can abandon the iteration after the first three items, and they don't have to "pay the cost" of generating the whole list.
To implement this kind of lazy list by hand would be troublesome. We would have to write two classes, one to represent the list by implementing IEnumerable<T>, and the other to represent an active enumeration operation by implementing IEnumerator<T>.
Iterator methods do all the hard work for us. We just write:
IEnumerable<int> GetNumbers(int stop)
{
for (int n = 0; n < stop; n++)
yield return n;
}
And the compiler converts this into two classes for us. Calling the method is equivalent to constructing an object of the class that represents the list.

Iterators are an abstraction that decouples the concept of position in a collection from the collection itself. The iterator is a separate object storing the necessary state to locate an item in the collection and move to the next item in the collection. I have seen collections that kept that state inside the collection (i.e. a current position), but it is often better to move that state to an external object. Among other things it enables you to have multiple iterators iterating the same collection.

Simple example : a function that generates a sequence of integers :
static IEnumerable<int> GetSequence(int fromValue, int toValue)
{
if (toValue >= fromValue)
{
for (int i = fromValue; i <= toValue; i++)
{
yield return i;
}
}
else
{
for (int i = fromValue; i >= toValue; i--)
{
yield return i;
}
}
}
To do it without an iterator, you would need to create an array then enumerate it...

Iterate through the students in a class
The Iterator design pattern provides
us with a common method of enumerating
a list of items or array, while hiding
the details of the list's
implementation. This provides a
cleaner use of the array object and
hides unneccessary information from
the client, ultimately leading to
better code-reuse, enhanced
maintainability, and fewer bugs. The
iterator pattern can enumerate the
list of items regardless of their
actual storage type.

Iterate through a set of homework questions.
But seriously, Iterators can provide a unified way to traverse the items in a collection regardless of the underlying data structure.
Read the first two paragraphs here for a little more info.

A couple of things they're great for:
a) For 'perceived performance' while maintaining code tidiness - the iteration of something separated from other processing logic.
b) When the number of items you're going to iterate through is not known.
Although both can be done through other means, with iterators the code can be made nicer and tidier as someone calling the iterator don't need to worry about how it finds the stuff to iterate through...
Real life example: enumerating directories and files, and finding the first [n] that fulfill some criteria, e.g. a file containing a certain string or sequence etc...

Beside everything else, to iterate through lazy-type sequences - IEnumerators. Each next element of such sequence may be evaluated/initialized upon iteration step which makes it possible to iterate through infinite sequences using finite amount of resources...

The canonical and simplest example is that it makes infinite sequences possible without the complexity of having to write the class to do that yourself:
// generate every prime number
public IEnumerator<int> GetPrimeEnumerator()
{
yield return 2;
var primes = new List<int>();
primesSoFar.Add(2);
Func<int, bool> IsPrime = n => primes.TakeWhile(
p => p <= (int)Math.Sqrt(n)).FirstOrDefault(p => n % p == 0) == 0;
for (int i = 3; true; i += 2)
{
if (IsPrime(i))
{
yield return i;
primes.Add(i);
}
}
}
Obviously this would not be truly infinite unless you used a BigInt instead of int but it gives you the idea.
Writing this code (or similar) for each generated sequence would be tedious and error prone. the iterators do that for you. If the above example seems too complex for you consider:
// generate every power of a number from start^0 to start^n
public IEnumerator<int> GetPowersEnumerator(int start)
{
yield return 1; // anything ^0 is 1
var x = start;
while(true)
{
yield return x;
x *= start;
}
}
They come at a cost though. Their lazy behaviour means you cannot spot common errors (null parameters and the like) until the generator is first consumed rather than created without writing wrapping functions to check first. The current implementation is also incredibly bad(1) if used recursively.
Wiriting enumerations over complex structures like trees and object graphs is much easier to write as the state maintenance is largely done for you, you must simply write code to visit each item and not worry about getting back to it.
I don't use this word lightly - a O(n) iteration can become O(N^2)

An iterator is an easy way of implementing the IEnumerator interface. Instead of making a class that has the methods and properties required for the interface, you just make a method that returns the values one by one and the compiler creates a class with the methods and properties needed to implement the interface.
If you for example have a large list of numbers, and you want to return a collection where each number is multiplied by two, you can make an iterator that returns the numbers instead of creating a copy of the list in memory:
public IEnumerable<int> GetDouble() {
foreach (int n in originalList) yield return n * 2;
}
In C# 3 you can do something quite similar using extension methods and lambda expressions:
originalList.Select(n => n * 2)
Or using LINQ:
from n in originalList select n * 2

IEnumerator<Question> myIterator = listOfStackOverFlowQuestions.GetEnumerator();
while (myIterator.MoveNext())
{
Question q;
q = myIterator.Current;
if (q.Pertinent == true)
PublishQuestion(q);
else
SendMessage(q.Author.EmailAddress, "Your question has been rejected");
}
foreach (Question q in listOfStackOverFlowQuestions)
{
if (q.Pertinent == true)
PublishQuestion(q);
else
SendMessage(q.Author.EmailAddress, "Your question has been rejected");
}

Some help understanding "yield"

In my everlasting quest to suck less I'm trying to understand the "yield" statement, but I keep encountering the same error.
The body of [someMethod] cannot be an iterator block because
'System.Collections.Generic.List< AClass>' is not an iterator interface type.
This is the code where I got stuck:
foreach (XElement header in headersXml.Root.Elements()){
yield return (ParseHeader(header));
}
What am I doing wrong? Can't I use yield in an iterator? Then what's the point?
In this example it said that List<ProductMixHeader> is not an iterator interface type.
ProductMixHeader is a custom class, but I imagine List is an iterator interface type, no?
--Edit--
Thanks for all the quick answers.
I know this question isn't all that new and the same resources keep popping up.
It turned out I was thinking I could return List<AClass> as a return type, but since List<T> isn't lazy, it cannot. Changing my return type to IEnumerable<T> solved the problem :D
A somewhat related question (not worth opening a new thread): is it worth giving IEnumerable<T> as a return type if I'm sure that 99% of the cases I'm going to go .ToList() anyway? What will the performance implications be?

A method using yield return must be declared as returning one of the following two interfaces:
IEnumerable<SomethingAppropriate>
IEnumerator<SomethingApropriate>
(thanks Jon and Marc for pointing out IEnumerator)
Example:
public IEnumerable<AClass> YourMethod()
{
foreach (XElement header in headersXml.Root.Elements())
{
yield return (ParseHeader(header));
}
}
yield is a lazy producer of data, only producing another item after the first has been retrieved, whereas returning a list will return everything in one go.
So there is a difference, and you need to declare the method correctly.
For more information, read Jon's answer here, which contains some very useful links.

It's a tricky topic. In a nutshell, it's an easy way of implementing IEnumerable and its friends. The compiler builds you a state machine, transforming parameters and local variables into instance variables in a new class. Complicated stuff.
I have a few resources on this:
Chapter 6 of C# in Depth (free download from that page)
Iterators, iterator blocks and data pipelines (article)
Iterator block implementation details (article)

"yield" creates an iterator block - a compiler generated class that can implement either IEnumerable[<T>] or IEnumerator[<T>]. Jon Skeet has a very good (and free) discussion of this in chapter 6 of C# in Depth.
But basically - to use "yield" your method must return an IEnumerable[<T>] or IEnumerator[<T>]. In this case:
public IEnumerable<AClass> SomeMethod() {
// ...
foreach (XElement header in headersXml.Root.Elements()){
yield return (ParseHeader(header));
}
}

List implements Ienumerable.
Here's an example that might shed some light on what you are trying to learn. I wrote this about 6 months
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace YieldReturnTest
{
public class PrimeFinder
{
private Boolean isPrime(int integer)
{
if (0 == integer)
return false;
if (3 > integer)
return true;
for (int i = 2; i < integer; i++)
{
if (0 == integer % i)
return false;
}
return true;
}
public IEnumerable<int> FindPrimes()
{
int i;
for (i = 1; i < 2147483647; i++)
{
if (isPrime(i))
{
yield return i;
}
}
}
}
class Program
{
static void Main(string[] args)
{
PrimeFinder primes = new PrimeFinder();
foreach (int i in primes.FindPrimes())
{
Console.WriteLine(i);
Console.ReadLine();
}
Console.ReadLine();
Console.ReadLine();
}
}
}

I highly recommend using Reflector to have a look at what yield actually does for you. You'll be able to see the full code of the class that the compiler generates for you when using yield, and I've found that people understand the concept much more quickly when they can see the low-level result (well, mid-level I guess).

To understand yield, you need to understand when to use IEnumerator and IEnumerable (because you have to use either of them). The following examples help you to understand the difference.
First, take a look at the following class, it implements two methods - one returning IEnumerator<int>, one returning IEnumerable<int>. I'll show you that there is a big difference in usage, although the code of the 2 methods is looking similar:
// 2 iterators, one as IEnumerator, one as IEnumerable
public class Iterator
{
public static IEnumerator<int> IterateOne(Func<int, bool> condition)
{
for(var i=1; condition(i); i++) { yield return i; }
}
public static IEnumerable<int> IterateAll(Func<int, bool> condition)
{
for(var i=1; condition(i); i++) { yield return i; }
}
}
Now, if you're using IterateOne you can do the following:
// 1. Using IEnumerator allows to get item by item
var i=Iterator.IterateOne(x => true); // iterate endless
// 1.a) get item by item
i.MoveNext(); Console.WriteLine(i.Current);
i.MoveNext(); Console.WriteLine(i.Current);
// 1.b) loop until 100
int j; while (i.MoveNext() && (j=i.Current)<=100) { Console.WriteLine(j); }
1.a) prints:
1
2
1.b) prints:
3
4
...
100
because it continues counting right after the 1.a) statements have been executed.
You can see that you can advance item by item using MoveNext().
In contrast, IterateAll allows you to use foreach and also LINQ statements for bigger comfort:
// 2. Using IEnumerable makes looping and LINQ easier
var k=Iterator.IterateAll(x => x<100); // limit iterator to 100
// 2.a) Use a foreach loop
foreach(var x in k){ Console.WriteLine(x); } // loop
// 2.b) LINQ: take 101..200 of endless iteration
var lst=Iterator.IterateAll(x=>true).Skip(100).Take(100).ToList(); // LINQ: take items
foreach(var x in lst){ Console.WriteLine(x); } // output list
2.a) prints:
1
2
...
99
2.b) prints:
101
102
...
200
Note: Since IEnumerator<T> and IEnumerable<T> are Generics, they can be used with any type. However, for simplicity I have used int in my examples for type T.
This means, you can use one of the return types IEnumerator<ProductMixHeader> or IEnumerable<ProductMixHeader> (the custom class you have mentioned in your question).
The type List<ProductMixHeader> does not implement any of these interfaces, which is the reason why you can't use it that way. But Example 2.b) is showing how you can create a list from it.
If you're creating a list by appending .ToList() then the implication is, that it will create a list of all elements in memory, while an IEnumerable allows lazy creation of its elements - in terms of performance, it means that elements are enumerated just in time - as late as possible, but as soon as you're using .ToList(), then all elements are created in memory. LINQ tries to optimize performance this way behind the scenes.
DotNetFiddle of all examples

#Ian P´s answer helped me a lot to understand yield and why it is used. One (major) use case for yield is in "foreach" loops after the "in" keyword not to return a fully completed list. Instead of returning a complete list at once, in each "foreach" loop only one item (the next item) is returned. So you will gain performance with yield in such cases.
I have rewritten #Ian P´s code for my better understanding to the following:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace YieldReturnTest
{
public class PrimeFinder
{
private Boolean isPrime(int integer)
{
if (0 == integer)
return false;
if (3 > integer)
return true;
for (int i = 2; i < integer; i++)
{
if (0 == integer % i)
return false;
}
return true;
}
public IEnumerable<int> FindPrimesWithYield()
{
int i;
for (i = 1; i < 2147483647; i++)
{
if (isPrime(i))
{
yield return i;
}
}
}
public IEnumerable<int> FindPrimesWithoutYield()
{
var primes = new List<int>();
int i;
for (i = 1; i < 2147483647; i++)
{
if (isPrime(i))
{
primes.Add(i);
}
}
return primes;
}
}
class Program
{
static void Main(string[] args)
{
PrimeFinder primes = new PrimeFinder();
Console.WriteLine("Finding primes until 7 with yield...very fast...");
foreach (int i in primes.FindPrimesWithYield()) // FindPrimesWithYield DOES NOT iterate over all integers at once, it returns item by item
{
if (i > 7)
{
break;
}
Console.WriteLine(i);
//Console.ReadLine();
}
Console.WriteLine("Finding primes until 7 without yield...be patient it will take lonkg time...");
foreach (int i in primes.FindPrimesWithoutYield()) // FindPrimesWithoutYield DOES iterate over all integers at once, it returns the complete list of primes at once
{
if (i > 7)
{
break;
}
Console.WriteLine(i);
//Console.ReadLine();
}
Console.ReadLine();
Console.ReadLine();
}
}
}

What does the method you're using this in look like? I don't think this can be used in just a loop by itself.
For example...
public IEnumerable<string> GetValues() {
foreach(string value in someArray) {
if (value.StartsWith("A")) { yield return value; }
}
}