We Keep Coding

How to avoid out of memory using trie by C#

how to write billions of data into a trie with less memory
I want to extract some infomation from news like company names,so I write billions of company names into a trie,but it needs much memory and throw out of memory exception,I don't know how to solve it,so anyone can help,thanks in advance.
public class Node
{
public char Value { get; set; }
public List<Node> Children { get; set; }
public int Depth { get; set; }
public string Code { get; set; }
public bool Terminal { get; set; }
public Node(char value, int depth)
{
Value = value;
Depth = depth;
Children = new List<Node>();
}
public Node FindChildNode(char c)
{
foreach (var child in Children)
if (child.Value == c)
return child;
return null;
}
}
public class Trie
{
private Node _root;
public Trie()
{
_root = new Node('^',0);
}
public Node Prefix(string s)
{
var currentNode = _root;
var result = currentNode;
foreach (var c in s)
{
currentNode = currentNode.FindChildNode(c);
if (currentNode == null)
break;
result = currentNode;
}
return result;
}
public void Insert(string randomLength,string code)
{
var commonPrefix = Prefix(randomLength);
var current = commonPrefix;
for (var i = current.Depth; i < s.Length; i++)
{
var newNode = new Node(s[i], current.Depth + 1);
if (i+1==s.Length)
{
newNode.Terminal = true;
newNode.Code = code;
}
current.Children.Add(newNode);
current = newNode;
}
}
}
Trie t=new Trie();
t.Insert("C","ABCG00DFD");
The aboved statement run 1000000000 Loops and the "C" can be replaced with different string with different length,as the loops increasing,it throw out of memory exception,so how to avoid or change it?

Have a go at this Trie and see if you can get it to work for what you need:
public class Trie : Dictionary<char, Trie>
{
public void Add(string value)
{
var c = String.IsNullOrEmpty(value) ? '\0' : value[0];
if (!this.ContainsKey(c))
{
this[c] = new Trie();
}
if (c != '\0')
{
this[c].Add(value.Substring(1));
}
}
}

How to free up memory from IEnumerable

I'm making a program that prints out all Nodes in a Collatz tree in C#.
This takes up a whole lot of memory, and outOfMemoryExceptions have been the word of the day. The base of the program is as follows:
var fileStream = File.Create("collatz.txt"); //just print into a textfile for testing
var writer = new StreamWriter(fileStream);
IEnumerable<Node> nodeList = new List<Node>()
{
new Node(1, null) //rootNode
};
Task flushTask = Task.CompletedTask;
while (nodeList.Any())
{
var tempList = CalcChildren(nodeList); //Return a IEnumerable of children of all the parents in the nodeList
await flushTask;
foreach (var node in tempList)
writer.WriteLine($"{node.Value} -> {node.Parent.Value}: {node.StepsFromRoot}");
flushTask = writer.FlushAsync();
nodeList = tempList;
}
writer.Close();
static IEnumerable<Node> CalcChildren(IEnumerable<Node> parents)
{
foreach(var parent in parents)
foreach (var child in CalcChildren(parent))
yield return child;
}
static IEnumerable<Node> CalcChildren(Node parent)
{
var multiValue = parent.CalcMultiplicationValue();
if (multiValue.HasValue)
{
var child = new Node(multiValue.Value, parent);
parent.MultiplicationChild = child;
yield return child;
}
var divValue = parent.CalcDivisionValue();
if (divValue.HasValue && divValue.Value!=1)
{
var child = new Node(divValue.Value, parent);
parent.DivisionChild = child;
yield return child;
}
}
I think herein lies the problem but for the sake of completeness, the Node class:
public class Node
{
public Node(int value, Node parent)
{
Value = value;
Parent = parent;
if (parent != null)
StepsFromRoot = parent.StepsFromRoot+1;
else
StepsFromRoot = 0;
}
public int Value { get; }
public Node Parent { get; set; }
public Node DivisionChild { get; set; }
public Node MultiplicationChild { get; set; }
public int StepsFromRoot { get; set; }
public int? CalcMultiplicationValue()
{
if(Value<=int.MaxValue/2)
return 2 * Value;
return null;
}
public int? CalcDivisionValue()
{
double newValue = (Value - 1) / 3.0;
if (newValue % 2 == 1 && newValue >= 1)
return (int)newValue;
return null;
}
}
I'm trying my best to drop as much memory as possible, but it doesn't seem to work. Around the 105th iteration this program takes up 4 GB of memory! I first thought it was because the buffer of my writer was taking up all the memory. Now I think it's because the collection of nodes isn't cleaned up by the garbage collector.
I tried to make it more minimal. But I'm not sure what to delete while keeping it complete. Thanks in advance!

How to loop recursive data type

I reading hierarchical data into a recursive data structure.
public class Items
{
public string Name { get; set; }
public List<Items> Children { get; set; }
}
So its similar to a tree. My problem is, I have no idea how I can loop over all elements or find a inner element with a specific Name. Since it can be very complex/deep I can not really work with nested loops since I don't know how deep it will get.
How can I have a loop over all elements in such a structure?

Since you need solution without recursion, here is one:
public Items FindByName(Items root, string targetName)
{
var stack = new Stack<Items>();
stack.Push(root);
Items node;
while (true)
{
node = stack.Pop();
if (node == null)
{
// not found ..
}
if (node.Name == targetName)
{
break;
}
foreach (var child in node.Children)
{
stack.Push(child);
}
}
return node;
}

void RecursiveMethod(Items items)
{
if (items.Children != null)
{
foreach (Items i in items.Children)
{
RecursiveMethod(i);
}
}
if (items.Name == "YourName")
{
// Do your stuff..
}
}

You need some traversal algorithm implementation.
There are several ones, either recursive, or non-recursive. It depends on your particular use-cases, which one to choose.
E.g., a kind of non-recursive, lazy width traversal:
public static class TreeVisitor
{
public static IEnumerable<TNodeType> WidthTraversal<TNodeType>(TNodeType root, Func<TNodeType, IEnumerable<TNodeType>> getChildNodesFunc)
where TNodeType : class
{
if (root == null)
{
throw new ArgumentNullException(nameof(root));
}
if (getChildNodesFunc == null)
{
throw new ArgumentNullException(nameof(getChildNodesFunc));
}
var visited = new HashSet<TNodeType>();
var queue = new Queue<TNodeType>();
yield return root;
visited.Add(root);
queue.Enqueue(root);
while (queue.Count > 0)
{
var parent = queue.Dequeue();
foreach (var child in getChildNodesFunc(parent))
{
if (child == default(TNodeType))
continue;
if (!visited.Contains(child))
{
yield return child;
visited.Add(child);
queue.Enqueue(child);
}
}
}
}
}
Usage:
var rootItem = new Items
{
Name = "Root",
Children = new List<Items>
{
new Items { Name = "Child1" },
new Items { Name = "Child2" },
// etc
}
};
foreach (var item in TreeVisitor.WidthTraversal(rootItem, _ => _.Children))
{
// ...
}

I would do it this way:
class Program
{
static void Main(string[] args)
{
List<Item> items = new List<Item>() { new Item { Name = "Pasta", Children = new List<Item>() { new Item { Name = "Pasta", Children = null } } } };
List<Item> pastas = GetItemsByName(items, "Pasta");
}
private static List<Item> GetItemsByName(List<Item> items, string name)
{
List<Item> found = new List<Item>();
foreach (Item item in items)
{
if (item.Name == name)
{
found.Add(item);
}
if (item.Children != null)
{
found.AddRange(GetItemsByName(item.Children, name));
}
}
return found;
}
}
public class Item
{
public string Name { get; set; }
public List<Item> Children { get; set; }
}

How to get parent node in a tree structure like this

How it would be possible to get a parent when tree structure is like this:
public class TreeModel
{
public int ID { get; set; }
public List<TreeModel> Children { get; set; }
}
Let's say we can't add a parent element item to this class (public TreeModel Parent { get; set; }).
Edit
How to get element m22 (ID=22) parent m2 (ID=2) from the m1? I thought we could iterate through m1 and somehow return parent when condition is right.
var m1 = new TreeModel() { ID = 1 };
var m2 = new TreeModel() { ID = 2 };
var m21 = new TreeModel() { ID = 21 };
var m22 = new TreeModel() { ID = 22 };
var m3 = new TreeModel() { ID = 3 };
m1.Children.Add(m2);
m2.Children.Add(m21);
m2.Children.Add(m22);
m1.Children.Add(m3);
var parent = m1.GetParent(p => p.ID == 22); //<-- How?

public IEnumerable<TreeModel> GetAllDescendants(IEnumerable<TreeModel> rootNodes)
{
var descendants = rootNodes.SelectMany(_ => GetAllDescendants(_.Children));
return rootNodes.Concat(descendants);
}
public static TreeModel GetParent(this TreeModel rootNode, Func<TreeModel, bool> childSelector)
{
var allNodes = GetAllDescendants(new [] { rootNode });
var parentsOfSelectedChildren = allNodes.Where(node => node.Children.Any(childSelector));
return parentsOfSelectedChildren.Single();
}
m1.GetParent(_ => _.ID == 22);
Obtain a flat list of all nodes
Search this list for the node whose direct children contains m22

Use this code pattern. It simplifies the code because you don't have to explicitly add nodes to the children and each node knows who its parent is and who its children are. Also it is all type safe.
class Program
{
static void Main(string[] args)
{
var m1=new TreeModel() { ID=1 };
var m2=new TreeModel(m1) { ID=2 };
var m21=new TreeModel(m2) { ID=21 };
var m22=new TreeModel(m2) { ID=22};
var m3=new TreeModel(m1) { ID=3 };
var item=m1.RecursiveFind((p) => p.ID==22);
var parent=item.Parent;
// parent.ID == 2
var root=item.Root;
// root.ID == 1;
}
}
public class TreeModel : Tree<TreeModel>
{
public int ID { get; set; }
public TreeModel() { }
public TreeModel(TreeModel parent) : base(parent) { }
}
public class Tree<T> where T : Tree<T>
{
protected Tree() : this(null) { }
protected Tree(T parent)
{
Parent=parent;
Children=new List<T>();
if(parent!=null)
{
parent.Children.Add(this as T);
}
}
public T Parent { get; set; }
public List<T> Children { get; set; }
public bool IsRoot { get { return Parent==null; } }
public T Root { get { return IsRoot?this as T:Parent.Root; } }
public T RecursiveFind(Predicate<T> check)
{
if(check(this as T)) return this as T;
foreach(var item in Children)
{
var result=item.RecursiveFind(check);
if(result!=null)
{
return result;
}
}
return null;
}
}
When you derive from Tree<T>, you create custom tree structures that you design what the node class is (TreeModel here) and how to handle parents, children and siblings if needed.

What about:
public class SaneTreeModel: TreeModel
{
public SaneTreeModel Parent { get; set; }
}
}

I would approach this by first finding the first element that satisfies the condition (ID == 22 in your example) and then finding the parent of this element. Not the best solution but maybe you will need them separately for something else.
public TreeModel GetParent(Func<TreeModel, bool> function)
{
return GetParent(Where(function));
}
private TreeModel GetParent(TreeModel treeModel)
{
if (Children == null) return null;
if (Children.Contains(treeModel)) return this;
foreach (TreeModel child in Children)
{
TreeModel result = child.GetParent(treeModel);
if (result != null)
return result;
}
return null;
}
private TreeModel Where(Func<TreeModel, bool> function)
{
if (Children == null) return null;
foreach (TreeModel child in Children)
{
if (function(child))
return child;
TreeModel result = child.Where(function);
if (result != null)
return result;
}
return null;
}
If you put this code block in your TreeModel class, the example you provided will return m2

Absolutely not, with a child node like that you can't get its parent. Simply because there isn't any reference to it.
To get the parent of the node, you have to either add the parent field or save the reference in somewhere else (by a variable or something).
EDIT
#Zulis If you search from the root node, you can definitely find the node you want. But as I said, with just the child node you can't do that.
But I think you should avoid searching because that would be slow

Recursive tree creation

I have a Node class in C# with the following properties:
public class Node
{
public int Id {get;set;}
public int? ParentId {get;set;}
public string Label {get;set;}
}
I have a TreeView control which provides the following method to create
a new node:
MyTreeView.CreateNode(key, label);
parent.Nodes.CreateNode(key, label);
If I want to add a new child node I need to use the second method otherwise the first one. Both returns an object of type TreeNode.
How would you create in C# a recursive function to populate the treeview considering that the root nodes have ParentId = null?
This is what I have done so far:
// create a list of root nodes
var roots = myList.Where(x => x.ParentId == null);
// send the roots to a recursive func
foreach(var root in roots)
{
AddNode(null,root,myList);
}
this is my recursive function:
private void AddNode(Node parent, Node current, IList<Node> items)
{
TreeNode treenode = null;
if(parent == null)
{
treenode = mytree.CreateNode(current.Id.ToString(), current.Label);
}else{
var parentnode = mytree.GetNode(parent.Id.ToString());
treenode = parentnode.Nodes.CreateNode(current.Id.ToString(), current.Label);
}
// call the recursion for the children
var children = items.Where(x => x.ParentId == current.Id);
foreach(var child in children)
{
AddNode(current, child, items);
}
}

If your tree view control is derived from System.Windows.Forms.TreeView you can replace
MyTreeView.CreateNode(key, label);
parent.Nodes.CreateNode(key, label);
with
MyTreeView.Nodes.Add(key, label);
parent.Nodes.Add(key, label);
So the call always goes to a Nodes collection which is of type TreeNodeCollection. Instead of your Node object you can now use the Nodes collection as parameter.
var roots = myList.Where(x => x.ParentId == null);
foreach (var root in roots)
{
AddNode(mytree.Nodes, root, myList);
}
private void AddNode(TreeNodeCollection nodes, Node current, IList<Node> items)
{
TreeNode treenode = nodes.Add(current.Id.ToString(), current.Label);
var children = items.Where(x => x.ParentId == current.Id);
foreach (var child in children)
{
AddNode(treenode.Nodes, child, items);
}
}
This has two benefits:
You don't need to lookup the parent each time.
You only have one call (TreeNodeCollection.Add).
However, if you can not access the TreeView.Nodes collection in the AddNode call for each root you will have to check for that at the top of the AddNode method.
var roots = myList.Where(x => x.ParentId == null);
foreach (var root in roots)
{
AddNode(null, root, myList);
}
private void AddNode(TreeNodeCollection nodes, Node current, IList<Node> items)
{
if (nodes == null)
{
nodes = myTree.Nodes;
}
...
}

Try this code:
var lookup = myList.ToLookup(n => n.ParentId.ToString());
Action<IEnumerable<TreeNode>> addChildren = null;
addChildren = tns =>
{
var query =
from tn in tns
from cn in lookup[tn.Name]
select tn.Nodes.CreateNode(cn.Id.ToString(), cn.Label);
var nodes = query.ToArray();
if (nodes.Length > 0)
{
addChildren(nodes);
}
};
addChildren(
lookup[null]
.Select(n =>
MyTreeView.CreateNode(n.Id.ToString(), n.Label)));
I couldn't fully test it, so you might need to change some of the code to make it work, but it should be fairly close.

I'd probably do something like this...
public class Node
{
public int Id { get; set; }
public int? ParentId { get; set; }
public string Label { get; set; }
public Node(int? parentId, int id, string label)
{
ParentId = parentId;
Id = id;
Label = label;
}
}
public class TreeNode : List<TreeNode>
{
public string Key { get; set; }
public string Label { get; set; }
public IEnumerable<TreeNode> Descendants
{
get
{
yield return this;
foreach (var child in this)
{
foreach (var descendant in child.Descendants)
{
yield return descendant;
}
}
}
}
public TreeNode(string key, string label)
{
Key = key;
Label = label;
}
public void CreateNode(int id, string label)
{
Add(new TreeNode(id.ToString(), label));
}
}
public class Tree
{
private TreeNode _root = new TreeNode(null, null);
public Tree(IEnumerable<Node> nodes)
{
nodes.ToList().ForEach(node => CreateNode(node.ParentId, node.Id, node.Label));
}
public void CreateNode(int? parentId, int id, string label)
{
if (parentId == null)
{
_root.CreateNode(id, label);
}
else
{
_root.Descendants.First(x => x.Key == parentId.ToString()).CreateNode(id, label);
}
}
public IEnumerable<TreeNode> Descendants => _root.Descendants;
}

Try this code:
Node{
Id, Label, List<Tree> Children
}
Tree GetTree(id){
var node=new Node();
node.Id=id;
node.Children=new List<Node>();
List<Node> children = db.Nodes.Where(x => x.ParentId==id);
foreach(child in children){
var childTree=GetTree(child.Id);
node.Children.Add(childTree);
}
return node;
}