Not sure how to call it, but say you have a class that looks like this:
class Person
{
public string Name;
public IEnumerable<Person> Friends;
}
You then have a person and you want to "unroll" this structure recursively so you end up with a single list of all people without duplicates.
How would you do this? I have already made something that seems to be working, but I am curious to see how others would do it and especially if there is something built-in to Linq you can use in a clever way to solve this little problem :)
Here is my solution:
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> subjects, Func<T, IEnumerable<T>> selector)
{
// Stop if subjects are null or empty
if(subjects == null)
yield break;
// For each subject
foreach(var subject in subjects)
{
// Yield it
yield return subject;
// Then yield all its decendants
foreach (var decendant in SelectRecursive(selector(subject), selector))
yield return decendant;
}
}
Would be used something like this:
var people = somePerson.SelectRecursive(x => x.Friends);
I don't believe there's anything built into LINQ to do this.
There's a problem with doing it recursively like this - you end up creating a large number of iterators. This can be quite inefficient if the tree is deep. Wes Dyer and Eric Lippert have both blogged about this.
You can remove this inefficiency by removing the direct recursion. For example:
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> subjects,
Func<T, IEnumerable<T>> selector)
{
if (subjects == null)
{
yield break;
}
Queue<T> stillToProcess = new Queue<T>(subjects);
while (stillToProcess.Count > 0)
{
T item = stillToProcess.Dequeue();
yield return item;
foreach (T child in selector(item))
{
stillToProcess.Enqueue(child);
}
}
}
This will also change the iteration order - it becomes breadth-first instead of depth-first; rewriting it to still be depth-first is tricky. I've also changed it to not use Any() - this revised version won't evaluate any sequence more than once, which can be handy in some scenarios. This does have one problem, mind you - it will take more memory, due to the queuing. We could probably alleviate this by storing a queue of iterators instead of items, but I'm not sure offhand... it would certainly be more complicated.
One point to note (also noted by ChrisW while I was looking up the blog posts :) - if you have any cycles in your friends list (i.e. if A has B, and B has A) then you'll recurse forever.
I found this question as I was looking for and thinking about a similar solution - in my case creating an efficient IEnumerable<Control> for ASP.NET UI controls. The recursive yield I had is fast but I knew that could have extra cost, since the deeper the control structure the longer it could take. Now I know this is O(n log n).
The solution given here provides some answer but, as discussed in the comments, it does change the order (which the OP did not care about). I realized that to preserve the order as given by the OP and as I needed, neither a simple Queue (as Jon used) nor Stack would work since all the parent objects would be yielded first and then any children after them (or vice-versa).
To resolve this and preserve the order I realized the solution would simply be to put the Enumerator itself on a Stack. To use the OPs original question it would look like this:
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> subjects, Func<T, IEnumerable<T>> selector)
{
if (subjects == null)
yield break;
var stack = new Stack<IEnumerator<T>>();
stack.Push(subjects.GetEnumerator());
while (stack.Count > 0)
{
var en = stack.Peek();
if (en.MoveNext())
{
var subject = en.Current;
yield return subject;
stack.Push(selector(subject).GetEnumerator());
}
else
{
stack.Pop().Dispose();
}
}
}
I use stack.Peek here to keep from having to push the same enumerator back on to the stack as this is likely to be the more frequent operation, expecting that enumerator to provide more than one item.
This creates the same number of enumerators as in the recursive version but will likely be fewer new objects than putting all the subjects in a queue or stack and continuing to add any descendant subjects. This is O(n) time as each enumerator stands on its own (in the recursive version an implicit call to one MoveNext executes MoveNext on the child enumerators to the current depth in the recursion stack).
You could use a non-recursive method like this as well:
HashSet<Person> GatherAll (Person p) {
Stack<Person> todo = new Stack<Person> ();
HashSet<Person> results = new HashSet<Person> ();
todo.Add (p); results.Add (p);
while (todo.Count > 0) {
Person p = todo.Pop ();
foreach (Person f in p.Friends)
if (results.Add (f)) todo.Add (f);
}
return results;
}
This should handle cycles properly as well. I am starting with a single person, but you could easily expand this to start with a list of persons.
Here's an implementation that:
Does a depth first recursive select,
Doesn't require double iteration of the child collections,
Doesn't use intermediate collections for the selected elements,
Doesn't handle cycles,
Can do it backwards.
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> rootItems, Func<T, IEnumerable<T>> selector)
{
return new RecursiveEnumerable<T>(rootItems, selector, false);
}
public static IEnumerable<T> SelectRecursiveReverse<T>(this IEnumerable<T> rootItems, Func<T, IEnumerable<T>> selector)
{
return new RecursiveEnumerable<T>(rootItems, selector, true);
}
class RecursiveEnumerable<T> : IEnumerable<T>
{
public RecursiveEnumerable(IEnumerable<T> rootItems, Func<T, IEnumerable<T>> selector, bool reverse)
{
_rootItems = rootItems;
_selector = selector;
_reverse = reverse;
}
IEnumerable<T> _rootItems;
Func<T, IEnumerable<T>> _selector;
bool _reverse;
public IEnumerator<T> GetEnumerator()
{
return new Enumerator(this);
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
class Enumerator : IEnumerator<T>
{
public Enumerator(RecursiveEnumerable<T> owner)
{
_owner = owner;
Reset();
}
RecursiveEnumerable<T> _owner;
T _current;
Stack<IEnumerator<T>> _stack = new Stack<IEnumerator<T>>();
public T Current
{
get
{
if (_stack == null || _stack.Count == 0)
throw new InvalidOperationException();
return _current;
}
}
public void Dispose()
{
_current = default(T);
if (_stack != null)
{
while (_stack.Count > 0)
{
_stack.Pop().Dispose();
}
_stack = null;
}
}
object System.Collections.IEnumerator.Current
{
get { return Current; }
}
public bool MoveNext()
{
if (_owner._reverse)
return MoveReverse();
else
return MoveForward();
}
public bool MoveForward()
{
// First time?
if (_stack == null)
{
// Setup stack
_stack = new Stack<IEnumerator<T>>();
// Start with the root items
_stack.Push(_owner._rootItems.GetEnumerator());
}
// Process enumerators on the stack
while (_stack.Count > 0)
{
// Get the current one
var se = _stack.Peek();
// Next please...
if (se.MoveNext())
{
// Store it
_current = se.Current;
// Get child items
var childItems = _owner._selector(_current);
if (childItems != null)
{
_stack.Push(childItems.GetEnumerator());
}
return true;
}
// Finished with the enumerator
se.Dispose();
_stack.Pop();
}
// Finished!
return false;
}
public bool MoveReverse()
{
// First time?
if (_stack == null)
{
// Setup stack
_stack = new Stack<IEnumerator<T>>();
// Start with the root items
_stack.Push(_owner._rootItems.Reverse().GetEnumerator());
}
// Process enumerators on the stack
while (_stack.Count > 0)
{
// Get the current one
var se = _stack.Peek();
// Next please...
if (se.MoveNext())
{
// Get child items
var childItems = _owner._selector(se.Current);
if (childItems != null)
{
_stack.Push(childItems.Reverse().GetEnumerator());
continue;
}
// Store it
_current = se.Current;
return true;
}
// Finished with the enumerator
se.Dispose();
_stack.Pop();
if (_stack.Count > 0)
{
_current = _stack.Peek().Current;
return true;
}
}
// Finished!
return false;
}
public void Reset()
{
Dispose();
}
}
}
use the Aggregate extension...
List<Person> persons = GetPersons();
List<Person> result = new List<Person>();
persons.Aggregate(result,SomeFunc);
private static List<Person> SomeFunc(List<Person> arg1,Person arg2)
{
arg1.Add(arg2)
arg1.AddRange(arg2.Persons);
return arg1;
}
Recursion is always fun. Perhaps you could simplify your code to:
public static IEnumerable<T> SelectRecursive<T>(this IEnumerable<T> subjects, Func<T, IEnumerable<T>> selector) {
// Stop if subjects are null or empty
if (subjects == null || !subjects.Any())
return Enumerable.Empty<T>();
// Gather a list of all (selected) child elements of all subjects
var subjectChildren = subjects.SelectMany(selector);
// Jump into the recursion for each of the child elements
var recursiveChildren = SelectRecursive(subjectChildren, selector);
// Combine the subjects with all of their (recursive child elements).
// The union will remove any direct parent-child duplicates.
// Endless loops due to circular references are however still possible.
return subjects.Union(recursiveChildren);
}
It will result in less duplicates than your original code. However their might still be duplicates causing an endless loop, the union will only prevent direct parent(s)-child(s) duplicates.
And the order of the items will be different from yours :)
Edit: Changed the last line of code to three statements and added a bit more documentation.
While its great to have IEnumerable when there might be a lot of data, its worth remembering the classic approach of recursively adding to a list.
That can be as simple as this (I've left out selector; just demonstrating recursively adding to an output list):
class Node
{
public readonly List<Node> Children = new List<Node>();
public List<Node> Flatten()
{
var all = new List<Node>();
Flatten(ref all);
return all;
}
public void Flatten(List<Node> all)
{
all.Add(this);
foreach (var child in Children)
child.Flatten(all);
}
}
usage:
Node rootNode = ...;
...
var all = rootNode.Flatten();
Related
I need to search a tree for data that could be anywhere in the tree. How can this be done with linq?
class Program
{
static void Main(string[] args) {
var familyRoot = new Family() {Name = "FamilyRoot"};
var familyB = new Family() {Name = "FamilyB"};
familyRoot.Children.Add(familyB);
var familyC = new Family() {Name = "FamilyC"};
familyB.Children.Add(familyC);
var familyD = new Family() {Name = "FamilyD"};
familyC.Children.Add(familyD);
//There can be from 1 to n levels of families.
//Search all children, grandchildren, great grandchildren etc, for "FamilyD" and return the object.
}
}
public class Family {
public string Name { get; set; }
List<Family> _children = new List<Family>();
public List<Family> Children {
get { return _children; }
}
}
That's an extension to It'sNotALie.s answer.
public static class Linq
{
public static IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten(c, selector))
.Concat(new[] { source });
}
}
Sample test usage:
var result = familyRoot.Flatten(x => x.Children).FirstOrDefault(x => x.Name == "FamilyD");
Returns familyD object.
You can make it work on IEnumerable<T> source too:
public static IEnumerable<T> Flatten<T>(this IEnumerable<T> source, Func<T, IEnumerable<T>> selector)
{
return source.SelectMany(x => Flatten(x, selector))
.Concat(source);
}
Another solution without recursion...
var result = FamilyToEnumerable(familyRoot)
.Where(f => f.Name == "FamilyD");
IEnumerable<Family> FamilyToEnumerable(Family f)
{
Stack<Family> stack = new Stack<Family>();
stack.Push(f);
while (stack.Count > 0)
{
var family = stack.Pop();
yield return family;
foreach (var child in family.Children)
stack.Push(child);
}
}
Simple:
familyRoot.Flatten(f => f.Children);
//you can do whatever you want with that sequence there.
//for example you could use Where on it and find the specific families, etc.
IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten(selector(c), selector))
.Concat(new[]{source});
}
So, the simplest option is to write a function that traverses your hierarchy and produces a single sequence. This then goes at the start of your LINQ operations, e.g.
IEnumerable<T> Flatten<T>(this T source)
{
foreach(var item in source) {
yield item;
foreach(var child in Flatten(item.Children)
yield child;
}
}
To call simply: familyRoot.Flatten().Where(n => n.Name == "Bob");
A slight alternative would give you a way to quickly ignore a whole branch:
IEnumerable<T> Flatten<T>(this T source, Func<T, bool> predicate)
{
foreach(var item in source) {
if (predicate(item)) {
yield item;
foreach(var child in Flatten(item.Children)
yield child;
}
}
Then you could do things like: family.Flatten(n => n.Children.Count > 2).Where(...)
I like Kenneth Bo Christensen's answer using stack, it works great, it is easy to read and it is fast (and doesn't use recursion).
The only unpleasant thing is that it reverses the order of child items (because stack is FIFO). If sort order doesn't matter to you then it's ok.
If it does, sorting can be achieved easily using selector(current).Reverse() in the foreach loop (the rest of the code is the same as in Kenneth's original post)...
public static IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
var stack = new Stack<T>();
stack.Push(source);
while (stack.Count > 0)
{
var current = stack.Pop();
yield return current;
foreach (var child in selector(current).Reverse())
stack.Push(child);
}
}
Well, I guess the way is to go with the technique of working with hierarchical structures:
You need an anchor to make
You need the recursion part
// Anchor
rootFamily.Children.ForEach(childFamily =>
{
if (childFamily.Name.Contains(search))
{
// Your logic here
return;
}
SearchForChildren(childFamily);
});
// Recursion
public void SearchForChildren(Family childFamily)
{
childFamily.Children.ForEach(_childFamily =>
{
if (_childFamily.Name.Contains(search))
{
// Your logic here
return;
}
SearchForChildren(_childFamily);
});
}
I have tried two of the suggested codes and made the code a bit more clear:
public static IEnumerable<T> Flatten1<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten1(c, selector)).Concat(new[] { source });
}
public static IEnumerable<T> Flatten2<T>(this T source, Func<T, IEnumerable<T>> selector)
{
var stack = new Stack<T>();
stack.Push(source);
while (stack.Count > 0)
{
var current = stack.Pop();
yield return current;
foreach (var child in selector(current))
stack.Push(child);
}
}
Flatten2() seems to be a little bit faster but its a close run.
Some further variants on the answers of It'sNotALie., MarcinJuraszek and DamienG.
First, the former two give a counterintuitive ordering. To get a nice tree-traversal ordering to the results, just invert the concatenation (put the "source" first).
Second, if you are working with an expensive source like EF, and you want to limit entire branches, Damien's suggestion that you inject the predicate is a good one and can still be done with Linq.
Finally, for an expensive source it may also be good to pre-select the fields of interest from each node with an injected selector.
Putting all these together:
public static IEnumerable<R> Flatten<T,R>(this T source, Func<T, IEnumerable<T>> children
, Func<T, R> selector
, Func<T, bool> branchpredicate = null
) {
if (children == null) throw new ArgumentNullException("children");
if (selector == null) throw new ArgumentNullException("selector");
var pred = branchpredicate ?? (src => true);
if (children(source) == null) return new[] { selector(source) };
return new[] { selector(source) }
.Concat(children(source)
.Where(pred)
.SelectMany(c => Flatten(c, children, selector, pred)));
}
I'm trying to write an extension method that is supposed to traverse an object graph and return all visited objects.
I'm not sure if my approach is the best, so please do comment on that. Also yield is frying my brain... I'm sure the answer is obvious :/
Model
public class MyClass
{
public MyClass Parent {get;set;}
}
Method
public static IEnumerable<T> SelectNested<T>
(this T source, Func<T, T> selector)
where T : class
{
yield return source;
var parent = selector(source);
if (parent == null)
yield break;
yield return SelectNestedParents(parent, selector).FirstOrDefault();
}
Usage
var list = myObject.SelectNested(x => x.Parent);
The problem
It's almost working. But it only visits 2 objects. It self and the parent.
So given this graph c -> b -> a starting from c. c, b is returned which is not quite what I wanted.
The result I'm looking for is b, c
In the last line of SelectNested you only return the first parent:
yield return SelectNestedParents(parent, selector).FirstOrDefault();
You have to return all parents:
foreach (var p in SelectNestedParents(parent, selector))
return p;
Instead of using recursion you can use iteration which probably is more efficient:
public static IEnumerable<T> SelectNested<T>(this T source, Func<T, T> selector)
where T : class {
var current = source;
while (current != null) {
yield return current;
current = selector(current);
}
}
The following code should work as expected:
public static IEnumerable<T> SelectNested<T>()
{
if (source != null){
yield return source;
var parent = selector(source);
// Result of the recursive call is IEnumerable<T>
// so you need to iterate over it and return its content.
foreach (var parent in (SelectNested(selector(source))))
{
yield return parent;
}
}
}
Strictly speaking, your class looks to be a list, not a graph, since selector returns only one object not an enumeration of them. Thus recursion is not necessary.
public static IEnumerable<T> SelectNested<T>(this T source, Func<T, T> selector)
where T : class
{
while (source != null)
{
yield return source;
source = selector(source);
}
}
I find myself regularly writing recursive IEnumerable<T> iterators to implement the same "Descendants" pattern as provided by, for example, XContainer.Descendants. The pattern I keep implementing is as follows, given a type Foo with a single-level iterator called Children:
public static IEnumerable<Foo> Descendants(this Foo root) {
foreach (var child in root.Children()) {
yield return child;
foreach (var subchild in child.Descendants()) {
yield return subchild;
}
}
}
This old StackOverflow question suggests the same pattern. But for some reason it feels weird to me to have to reference three levels of heirarchy (root, child, and subchild). Can this fundamental depth-first recursion pattern be further reduced? Or is this an algorithmic primitive of sorts?
The best I can come up with is to abstract the pattern to a generic extension. This doesn't reduce the logic of the iterator pattern presented above, but it does remove the requirement of defining a Descendants method for multiple specific classes. On the downside, this adds an extension method to Object itself, which is a little smelly:
public static IEnumerable<T> SelectRecurse<T>(
this T root, Func<T, IEnumerable<T>> enumerator) {
foreach (T item in enumerator(root))
{
yield return item;
foreach (T subitem in item.SelectRecurse(enumerator))
{
yield return subitem;
}
}
}
// Now we can just write:
foreach(var item in foo.SelectRecurse(f => f.Children())) { /* do stuff */ }
You can use an explicit stack, rather than implicitly using the thread's call stack, to store the data that you are using. This can even be generalized to a Traverse method that just accepts a delegate to represent the "get my children" call:
public static IEnumerable<T> Traverse<T>(
this IEnumerable<T> source
, Func<T, IEnumerable<T>> childrenSelector)
{
var stack = new Stack<T>(source);
while (stack.Any())
{
var next = stack.Pop();
yield return next;
foreach (var child in childrenSelector(next))
stack.Push(child);
}
}
Because this isn't recursive, and thus isn't creating the state machines constantly, it will perform quite a bit better.
Side note, if you want a Breath First Search just use a Queue instead of a Stack. If you want a Best First Search use a priority queue.
To ensure that siblings are returned in the same order as they are returned from the selecor's order, rather than the reverse, just add a Reverse call to the result of childrenSelector.
I think this is a good question. The best explanation I have for why you need two loops: We need to recognize the fact that each item is converted to become multiple items (itself, and all its descendants). This means that we do not map one-to-one (like Select) but one-to-many (SelectMany).
We could write it like this:
public static IEnumerable<Foo> Descendants(this IEnumerable<Foo> items) {
foreach (var item in items) {
yield return item;
foreach (var subitem in item.Children().Descendants())
yield return subitem;
}
}
Or like this:
public static IEnumerable<Foo> Descendants(Foo root) {
var children = root.Children();
var subchildren = children.SelectMany(c => c.Descendants());
return children.Concat(subchildren);
}
Or like this:
public static IEnumerable<Foo> Descendants(this IEnumerable<Foo> items) {
var children = items.SelectMany(c => c.Descendants());
return items.Concat(children);
}
The versions taking an IEnumerable<Foo> must be invoked on root.Children().
I think all of these rewrites expose a different way of looking at the problem. On the other hand, they all have two nested loops. The loops can be hidden in helper functions but they still exist.
I would manage this with a List:
public static IEnumerable<Foo> Descendants(this Foo root) {
List<Foo> todo = new List<Foo>();
todo.AddRange(root.Children());
while(todo.Count > 0)
{
var first = todo[0];
todo.RemoveAt(0);
todo.InsertRange(0,first.Children());
yield return first;
}
}
Not recursive, so shouldn't blow the stack. You just always add more work for yourself onto the front of the list and so you achieve the depth-first traversal.
Both Damien_the_Unbeliever and Servy have presented versions of an algorithm that avoid creating a recursive call stack by using collections of one type or another. Damien's use of a List could cause poor performance to inserts at the head of the list, while Servy's use a of stack will cause nested elements to be returned in reverse order. I believe manually implementing a one-way linked list will maintain Servy's performance while still returning all the items in the original order. The only tricky part is initializing the first ForwardLinks by iterating the root. To keep Traverse clean I moved that to a constructor on ForwardLink.
public static IEnumerable<T> Traverse<T>(
this T root,
Func<T, IEnumerable<T>> childrenSelector) {
var head = new ForwardLink<T>(childrenSelector(root));
if (head.Value == null) yield break; // No items from root iterator
while (head != null)
{
var headValue = head.Value;
var localTail = head;
var second = head.Next;
// Insert new elements immediately behind head.
foreach (var child in childrenSelector(headValue))
localTail = localTail.Append(child);
// Splice on the old tail, if there was one
if (second != null) localTail.Next = second;
// Pop the head
yield return headValue;
head = head.Next;
}
}
public class ForwardLink<T> {
public T Value { get; private set; }
public ForwardLink<T> Next { get; set; }
public ForwardLink(T value) { Value = value; }
public ForwardLink(IEnumerable<T> values) {
bool firstElement = true;
ForwardLink<T> tail = null;
foreach (T item in values)
{
if (firstElement)
{
Value = item;
firstElement = false;
tail = this;
}
else
{
tail = tail.Append(item);
}
}
}
public ForwardLink<T> Append(T value) {
return Next = new ForwardLink<T>(value);
}
}
I propose a different version, without using yield:
public abstract class RecursiveEnumerator : IEnumerator {
public RecursiveEnumerator(ICollection collection) {
this.collection = collection;
this.enumerator = collection.GetEnumerator();
}
protected abstract ICollection GetChildCollection(object item);
public bool MoveNext() {
if (enumerator.Current != null) {
ICollection child_collection = GetChildCollection(enumerator.Current);
if (child_collection != null && child_collection.Count > 0) {
stack.Push(enumerator);
enumerator = child_collection.GetEnumerator();
}
}
while (!enumerator.MoveNext()) {
if (stack.Count == 0) return false;
enumerator = stack.Pop();
}
return true;
}
public virtual void Dispose() { }
public object Current { get { return enumerator.Current; } }
public void Reset() {
stack.Clear();
enumerator = collection.GetEnumerator();
}
private IEnumerator enumerator;
private Stack<IEnumerator> stack = new Stack<IEnumerator>();
private ICollection collection;
}
Usage example
public class RecursiveControlEnumerator : RecursiveEnumerator, IEnumerator {
public RecursiveControlEnumerator(Control.ControlCollection controlCollection)
: base(controlCollection) { }
protected override ICollection GetChildCollection(object c) {
return (c as Control).Controls;
}
}
To expand on my comment, this should work:
public static IEnumerable<Foo> Descendants(this Foo node)
{
yield return node; // return branch nodes
foreach (var child in node.Children())
foreach (var c2 in child.Descendants())
yield return c2; // return leaf nodes
}
That should will return all branch nodes and leaf nodes. If you only want to return leaf nodes, remove the first yield return.
In response to your question, yes it is an algorithmic primitive, because you definitely need to call node.Children(), and you definitely need to call child.Descendants() on each child. I agree that it seems odd having two "foreach" loops, but the second one is actually just continuing the overall enumeration, not iterating the children.
Try this:
private static IEnumerable<T> Descendants<T>(
this IEnumerable<T> children, Func<T, IEnumerable<T>> enumerator)
{
Func<T, IEnumerable<T>> getDescendants =
child => enumerator(child).Descendants(enumerator);
Func<T, IEnumerable<T>> getChildWithDescendants =
child => new[] { child }.Concat(getDescendants(child));
return children.SelectMany(getChildWithDescendants);
}
Or if you prefer the non Linq variant:
private static IEnumerable<T> Descendants<T>(
this IEnumerable<T> children, Func<T, IEnumerable<T>> enumerator)
{
foreach (var child in children)
{
yield return child;
var descendants = enumerator(child).Descendants(enumerator);
foreach (var descendant in descendants)
{
yield return descendant;
}
}
}
And call it like:
root.Children().Descendants(f => f.Children())
I need to search a tree for data that could be anywhere in the tree. How can this be done with linq?
class Program
{
static void Main(string[] args) {
var familyRoot = new Family() {Name = "FamilyRoot"};
var familyB = new Family() {Name = "FamilyB"};
familyRoot.Children.Add(familyB);
var familyC = new Family() {Name = "FamilyC"};
familyB.Children.Add(familyC);
var familyD = new Family() {Name = "FamilyD"};
familyC.Children.Add(familyD);
//There can be from 1 to n levels of families.
//Search all children, grandchildren, great grandchildren etc, for "FamilyD" and return the object.
}
}
public class Family {
public string Name { get; set; }
List<Family> _children = new List<Family>();
public List<Family> Children {
get { return _children; }
}
}
That's an extension to It'sNotALie.s answer.
public static class Linq
{
public static IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten(c, selector))
.Concat(new[] { source });
}
}
Sample test usage:
var result = familyRoot.Flatten(x => x.Children).FirstOrDefault(x => x.Name == "FamilyD");
Returns familyD object.
You can make it work on IEnumerable<T> source too:
public static IEnumerable<T> Flatten<T>(this IEnumerable<T> source, Func<T, IEnumerable<T>> selector)
{
return source.SelectMany(x => Flatten(x, selector))
.Concat(source);
}
Another solution without recursion...
var result = FamilyToEnumerable(familyRoot)
.Where(f => f.Name == "FamilyD");
IEnumerable<Family> FamilyToEnumerable(Family f)
{
Stack<Family> stack = new Stack<Family>();
stack.Push(f);
while (stack.Count > 0)
{
var family = stack.Pop();
yield return family;
foreach (var child in family.Children)
stack.Push(child);
}
}
Simple:
familyRoot.Flatten(f => f.Children);
//you can do whatever you want with that sequence there.
//for example you could use Where on it and find the specific families, etc.
IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten(selector(c), selector))
.Concat(new[]{source});
}
So, the simplest option is to write a function that traverses your hierarchy and produces a single sequence. This then goes at the start of your LINQ operations, e.g.
IEnumerable<T> Flatten<T>(this T source)
{
foreach(var item in source) {
yield item;
foreach(var child in Flatten(item.Children)
yield child;
}
}
To call simply: familyRoot.Flatten().Where(n => n.Name == "Bob");
A slight alternative would give you a way to quickly ignore a whole branch:
IEnumerable<T> Flatten<T>(this T source, Func<T, bool> predicate)
{
foreach(var item in source) {
if (predicate(item)) {
yield item;
foreach(var child in Flatten(item.Children)
yield child;
}
}
Then you could do things like: family.Flatten(n => n.Children.Count > 2).Where(...)
I like Kenneth Bo Christensen's answer using stack, it works great, it is easy to read and it is fast (and doesn't use recursion).
The only unpleasant thing is that it reverses the order of child items (because stack is FIFO). If sort order doesn't matter to you then it's ok.
If it does, sorting can be achieved easily using selector(current).Reverse() in the foreach loop (the rest of the code is the same as in Kenneth's original post)...
public static IEnumerable<T> Flatten<T>(this T source, Func<T, IEnumerable<T>> selector)
{
var stack = new Stack<T>();
stack.Push(source);
while (stack.Count > 0)
{
var current = stack.Pop();
yield return current;
foreach (var child in selector(current).Reverse())
stack.Push(child);
}
}
Well, I guess the way is to go with the technique of working with hierarchical structures:
You need an anchor to make
You need the recursion part
// Anchor
rootFamily.Children.ForEach(childFamily =>
{
if (childFamily.Name.Contains(search))
{
// Your logic here
return;
}
SearchForChildren(childFamily);
});
// Recursion
public void SearchForChildren(Family childFamily)
{
childFamily.Children.ForEach(_childFamily =>
{
if (_childFamily.Name.Contains(search))
{
// Your logic here
return;
}
SearchForChildren(_childFamily);
});
}
I have tried two of the suggested codes and made the code a bit more clear:
public static IEnumerable<T> Flatten1<T>(this T source, Func<T, IEnumerable<T>> selector)
{
return selector(source).SelectMany(c => Flatten1(c, selector)).Concat(new[] { source });
}
public static IEnumerable<T> Flatten2<T>(this T source, Func<T, IEnumerable<T>> selector)
{
var stack = new Stack<T>();
stack.Push(source);
while (stack.Count > 0)
{
var current = stack.Pop();
yield return current;
foreach (var child in selector(current))
stack.Push(child);
}
}
Flatten2() seems to be a little bit faster but its a close run.
Some further variants on the answers of It'sNotALie., MarcinJuraszek and DamienG.
First, the former two give a counterintuitive ordering. To get a nice tree-traversal ordering to the results, just invert the concatenation (put the "source" first).
Second, if you are working with an expensive source like EF, and you want to limit entire branches, Damien's suggestion that you inject the predicate is a good one and can still be done with Linq.
Finally, for an expensive source it may also be good to pre-select the fields of interest from each node with an injected selector.
Putting all these together:
public static IEnumerable<R> Flatten<T,R>(this T source, Func<T, IEnumerable<T>> children
, Func<T, R> selector
, Func<T, bool> branchpredicate = null
) {
if (children == null) throw new ArgumentNullException("children");
if (selector == null) throw new ArgumentNullException("selector");
var pred = branchpredicate ?? (src => true);
if (children(source) == null) return new[] { selector(source) };
return new[] { selector(source) }
.Concat(children(source)
.Where(pred)
.SelectMany(c => Flatten(c, children, selector, pred)));
}
Provided items is the result of a LINQ expression:
var items = from item in ItemsSource.RetrieveItems()
where ...
Suppose generation of each item takes some non-negligeble time.
Two modes of operation are possible:
Using foreach would allow to start working with items in the beginning of the collection much sooner than whose in the end become available. However if we wanted to later process the same collection again, we'll have to copy save it:
var storedItems = new List<Item>();
foreach(var item in items)
{
Process(item);
storedItems.Add(item);
}
// Later
foreach(var item in storedItems)
{
ProcessMore(item);
}
Because if we'd just made foreach(... in items) then ItemsSource.RetrieveItems() would get called again.
We could use .ToList() right upfront, but that would force us wait for the last item to be retrieved before we could start processing the first one.
Question: Is there an IEnumerable implementation that would iterate first time like regular LINQ query result, but would materialize in process so that second foreach would iterate over stored values?
A fun challenge so I have to provide my own solution. So fun in fact that my solution now is in version 3. Version 2 was a simplification I made based on feedback from Servy. I then realized that my solution had huge drawback. If the first enumeration of the cached enumerable didn't complete no caching would be done. Many LINQ extensions like First and Take will only enumerate enough of the enumerable to get the job done and I had to update to version 3 to make this work with caching.
The question is about subsequent enumerations of the enumerable which does not involve concurrent access. Nevertheless I have decided to make my solution thread safe. It adds some complexity and a bit of overhead but should allow the solution to be used in all scenarios.
public static class EnumerableExtensions {
public static IEnumerable<T> Cached<T>(this IEnumerable<T> source) {
if (source == null)
throw new ArgumentNullException("source");
return new CachedEnumerable<T>(source);
}
}
class CachedEnumerable<T> : IEnumerable<T> {
readonly Object gate = new Object();
readonly IEnumerable<T> source;
readonly List<T> cache = new List<T>();
IEnumerator<T> enumerator;
bool isCacheComplete;
public CachedEnumerable(IEnumerable<T> source) {
this.source = source;
}
public IEnumerator<T> GetEnumerator() {
lock (this.gate) {
if (this.isCacheComplete)
return this.cache.GetEnumerator();
if (this.enumerator == null)
this.enumerator = source.GetEnumerator();
}
return GetCacheBuildingEnumerator();
}
public IEnumerator<T> GetCacheBuildingEnumerator() {
var index = 0;
T item;
while (TryGetItem(index, out item)) {
yield return item;
index += 1;
}
}
bool TryGetItem(Int32 index, out T item) {
lock (this.gate) {
if (!IsItemInCache(index)) {
// The iteration may have completed while waiting for the lock.
if (this.isCacheComplete) {
item = default(T);
return false;
}
if (!this.enumerator.MoveNext()) {
item = default(T);
this.isCacheComplete = true;
this.enumerator.Dispose();
return false;
}
this.cache.Add(this.enumerator.Current);
}
item = this.cache[index];
return true;
}
}
bool IsItemInCache(Int32 index) {
return index < this.cache.Count;
}
IEnumerator IEnumerable.GetEnumerator() {
return GetEnumerator();
}
}
The extension is used like this (sequence is an IEnumerable<T>):
var cachedSequence = sequence.Cached();
// Pulling 2 items from the sequence.
foreach (var item in cachedSequence.Take(2))
// ...
// Pulling 2 items from the cache and the rest from the source.
foreach (var item in cachedSequence)
// ...
// Pulling all items from the cache.
foreach (var item in cachedSequence)
// ...
There is slight leak if only part of the enumerable is enumerated (e.g. cachedSequence.Take(2).ToList(). The enumerator that is used by ToList will be disposed but the underlying source enumerator is not disposed. This is because the first 2 items are cached and the source enumerator is kept alive should requests for subsequent items be made. In that case the source enumerator is only cleaned up when eligigble for garbage Collection (which will be the same time as the possibly large cache).
Take a look at the Reactive Extentsions library - there is a MemoizeAll() extension which will cache the items in your IEnumerable once they're accessed, and store them for future accesses.
See this blog post by Bart De Smet for a good read on MemoizeAll and other Rx methods.
Edit: This is actually found in the separate Interactive Extensions package now - available from NuGet or Microsoft Download.
public static IEnumerable<T> SingleEnumeration<T>(this IEnumerable<T> source)
{
return new SingleEnumerator<T>(source);
}
private class SingleEnumerator<T> : IEnumerable<T>
{
private CacheEntry<T> cacheEntry;
public SingleEnumerator(IEnumerable<T> sequence)
{
cacheEntry = new CacheEntry<T>(sequence.GetEnumerator());
}
public IEnumerator<T> GetEnumerator()
{
if (cacheEntry.FullyPopulated)
{
return cacheEntry.CachedValues.GetEnumerator();
}
else
{
return iterateSequence<T>(cacheEntry).GetEnumerator();
}
}
IEnumerator IEnumerable.GetEnumerator()
{
return this.GetEnumerator();
}
}
private static IEnumerable<T> iterateSequence<T>(CacheEntry<T> entry)
{
using (var iterator = entry.CachedValues.GetEnumerator())
{
int i = 0;
while (entry.ensureItemAt(i) && iterator.MoveNext())
{
yield return iterator.Current;
i++;
}
}
}
private class CacheEntry<T>
{
public bool FullyPopulated { get; private set; }
public ConcurrentQueue<T> CachedValues { get; private set; }
private static object key = new object();
private IEnumerator<T> sequence;
public CacheEntry(IEnumerator<T> sequence)
{
this.sequence = sequence;
CachedValues = new ConcurrentQueue<T>();
}
/// <summary>
/// Ensure that the cache has an item a the provided index. If not, take an item from the
/// input sequence and move to the cache.
///
/// The method is thread safe.
/// </summary>
/// <returns>True if the cache already had enough items or
/// an item was moved to the cache,
/// false if there were no more items in the sequence.</returns>
public bool ensureItemAt(int index)
{
//if the cache already has the items we don't need to lock to know we
//can get it
if (index < CachedValues.Count)
return true;
//if we're done there's no race conditions hwere either
if (FullyPopulated)
return false;
lock (key)
{
//re-check the early-exit conditions in case they changed while we were
//waiting on the lock.
//we already have the cached item
if (index < CachedValues.Count)
return true;
//we don't have the cached item and there are no uncached items
if (FullyPopulated)
return false;
//we actually need to get the next item from the sequence.
if (sequence.MoveNext())
{
CachedValues.Enqueue(sequence.Current);
return true;
}
else
{
FullyPopulated = true;
return false;
}
}
}
}
So this has been edited (substantially) to support multithreaded access. Several threads can ask for items, and on an item by item basis, they will be cached. It doesn't need to wait for the entire sequence to be iterated for it to return cached values. Below is a sample program that demonstrates this:
private static IEnumerable<int> interestingIntGenertionMethod(int maxValue)
{
for (int i = 0; i < maxValue; i++)
{
Thread.Sleep(1000);
Console.WriteLine("actually generating value: {0}", i);
yield return i;
}
}
public static void Main(string[] args)
{
IEnumerable<int> sequence = interestingIntGenertionMethod(10)
.SingleEnumeration();
int numThreads = 3;
for (int i = 0; i < numThreads; i++)
{
int taskID = i;
Task.Factory.StartNew(() =>
{
foreach (int value in sequence)
{
Console.WriteLine("Task: {0} Value:{1}",
taskID, value);
}
});
}
Console.WriteLine("Press any key to exit...");
Console.ReadKey(true);
}
You really need to see it run to understand the power here. As soon as a single thread forces the next actual values to be generated all of the remaining threads can immediately print that generated value, but they will all be waiting if there are no uncached values for that thread to print. (Obviously thread/threadpool scheduling may result in one task taking longer to print it's value than needed.)
There have already been posted thread-safe implementations of the Cached/SingleEnumeration operator by Martin Liversage and Servy respectively, and the thread-safe Memoise operator from the System.Interactive package is also available. In case thread-safety is not a requirement, and paying the cost of thread-synchronization is undesirable, there are answers offering unsynchronized ToCachedEnumerable implementations in this question. All these implementations have in common that they are based on custom types. My challenge was to write a similar not-synchronized operator in a single self-contained extension method (no strings attached). Here is my implementation:
public static IEnumerable<T> MemoiseNotSynchronized<T>(this IEnumerable<T> source)
{
// Argument validation omitted
IEnumerator<T> enumerator = null;
List<T> buffer = null;
return Implementation();
IEnumerable<T> Implementation()
{
if (buffer != null && enumerator == null)
{
// The source has been fully enumerated
foreach (var item in buffer) yield return item;
yield break;
}
enumerator ??= source.GetEnumerator();
buffer ??= new();
for (int i = 0; ; i = checked(i + 1))
{
if (i < buffer.Count)
{
yield return buffer[i];
}
else if (enumerator.MoveNext())
{
Debug.Assert(buffer.Count == i);
var current = enumerator.Current;
buffer.Add(current);
yield return current;
}
else
{
enumerator.Dispose(); enumerator = null;
yield break;
}
}
}
}
Usage example:
IEnumerable<Point> points = GetPointsFromDB().MemoiseNotSynchronized();
// Enumerate the 'points' any number of times, on a single thread.
// The data will be fetched from the DB only once.
// The connection with the DB will open when the 'points' is enumerated
// for the first time, partially or fully.
// The connection will stay open until the 'points' is enumerated fully
// for the first time.
Testing the MemoiseNotSynchronized operator on Fiddle.