We Keep Coding

Entity Framework performance issues adding child to list

I am working on a project where we are using Entity Framework 6.1.3. Right now we are experiencing pretty big performance issues when adding a child object to a list of the parent entity (see code sample below).
We are using lazy-loading, so what I noticed is that everything works fine until we call _parent.Children.Add(child); since it seems to load all children from the database just to be able to add a new one. Since some of our parent objects have about 50,000 children, this is delaying this simple insert call by 7-8 seconds and sometimes even causing timeouts.
To me it doesn't really make sense for Entity Framework to load all children just in order to add one, so is there a way I can avoid this or is this an Entity Framework design-flaw and should we find a workaround?
I'd obviously like to find a solution for this and would prefer not having to implement pure ADO queries for this one problem.
Thanks!
public class Parent
{
public Guid Id { get; set; }
public virtual ICollection<Child> Children { get; set; }
}
public class Child
{
public Guid Id { get; set; }
}
public class ParentAggregate
{
private readonly Parent _state;
public ParentAggregate(Parent state)
{
_state = state;
}
public void AddChild(Guid id)
{
var child = new Child { Id = id };
_state.Children.Add(child);
}
}

To me it doesn't really make sense for Entity Framework to load all children just in order to add one
The lazy loading occurs the first time you access a navigation property through its getter. And the sample code
_parent.Children.Add(child);
consists of two operations:
(1) retrieve the Children property (through the property getter!):
var children = _parent.Children;
(2) perform some operation on it (call Add method in this case):
children.Add(child);
The lazy loading occurs because of the operation (1). As you can see, EF has nothing to do with that because it has no control over it. And there is no way to know what you are going to do with that property value - enumerate it, take a count or use Add, Remove etc. methods.
Here are some solutions.
First, why using lazy loading at all? It has so many side effects and inefficiencies, and all they can easily be solved by EF provided out of the box eager loading via Include methods. That's why by default EF Core ("the future of EF") does not use lazy loading by default and requires a special package and procedure for enabling it.
Second, if you insist using lazy loading, then you have the following two options:
(A) Disable lazy loading during the data modifications (requires access to/control of the DbContext instance):
dbContext.Configuration.LazyLoadingEnabled = false;
_parent.Children.Add(child);
dbContext.Configuration.LazyLoadingEnabled = true;
This also requires the collection property to be initialized in order to avoid NRE.
(B) Use explicit backing field and provide some direct access to it (to avoid triggering the lazy load by property accessor). For instance:
public class Parent
{
public Guid Id { get; set; }
private ICollection<Child> children;
public virtual ICollection<Child> Children { get => children; set => children = value; }
public void Add(Child child)
{
// use the backing field directly
if (children == null) children = new HashSet<Child>();
children.Add(child);
}
}

Defensive Code to prevent Infinite Recursion in Parent/Child Hierarchy

Given an Object as Such
public class Thing
{
public Thing() { this.children = new List<Thing>();}
public int Id {get; set;}
public string Name {get; set;}
public List<Thing> children{ get; set;}
public string ToString(int level = 0)
{
//Level is added purely to add a visual hierarchy
var sb = new StringBuilder();
sb.Append(new String('-',level));
sb.AppendLine($"id:{Id} Name:{Name}");
foreach(var child in children)
{
sb.Append(child.ToString(level + 1));
}
return sb.ToString();
}
}
and if used (abused!?) in such a way
public static void Main()
{
var root = new Thing{Id = 1,Name = "Thing1"};
var thing2 = new Thing{Id = 2,Name = "Thing2"};
var thing3 = new Thing{Id = 3,Name = "Thing3"};
root.children.Add(thing2);
thing2.children.Add(thing3);
thing3.children.Add(root); //problem is here
Console.WriteLine(root.ToString());
}
how does one be defensive about this kind of scenario.
This code as it stands produces a stackoverflow, infinite recursion, or memory exceeded error.
In a (IIS) website this was causing the w3 worker processes to crash, and eventually the app pool to shut down (Rapid-Fail Protection)
The code above is indicative only to reproduce the problem. In the actual scenario, the structure is coming from a database with Id and ParentId.
Database table structure similar to
CREATE TABLE Thing(
Id INT NOT NULL PRIMARY KEY,
Name NVARCHAR(255) NOT NULL,
ParentThingId INT NULL //References self
)
The issue is that the creation of the 'things' by users is not preventing a incestuous relationship (i.e. a Parent could have children (who could have children etc.... that one eventually points at the parent again). One could put a constraint on the db to prevent the thing not being its own parent (makes sense), but depending on depth this could get ugly, and there is some argument that a circular reference may be required (we are still debating this....)
So arguably the structures can be circular, but if you want to render this kind of structure on a web page say as a <ul><li><a> tag kind of thing in a parent/child menu, how does one become proactive about dealing with this user generated data issue in code?
.NET fiddle here

One way would be to include a collection of visited nodes in the recursive call. If visited before you are in a cycle.
public string ToString(int level = 0, HashSet<int> visited)
{
foreach(var child in children)
{
if(visited.Add(child.Id))
sb.Append(child.ToString(level + 1, visited));
else
//Handle the case when a cycle is detected.
}
return sb.ToString();
}

You can unfold the tree structure by putting each element on a stack or queue and popping items of there while the collection has items. In the while loop you put the children of each item on the queue.
If you care about the level of the item in the tree you need can use a helper object that stores that.
Edit:
While unfolding the tree you can put each item on a new list and use that as reference for circular problems.

If you can a) eliminate that possibility of wanting to have circular references and b) guarantee that all children are already known of when that parent is created, its a great opportunity to make children an immutable collection that's only set via the constructor.
That gives you a class that, by structural recursion, you know cannot contain any loops, no matter how big the overall structure is. Something like:
public sealed class Thing
{
public Thing(IEnumerable<Thing> children) {
this._children = children.ToList().AsReadOnly();
}
private readonly ReadOnlyCollection<Thing> _children;
public int Id {get; set;}
public string Name {get; set;}
public IEnumerable<Thing> children {
get {
return _children;
}
}
public string ToString(int level = 0)
{
//Level is added purely to add a visual hierarchy
var sb = new StringBuilder();
sb.Append(new String('-',level));
sb.AppendLine($"id:{Id} Name:{Name}");
foreach(var child in children)
{
sb.Append(child.ToString(level + 1));
}
return sb.ToString();
}
}
Now, of course, those conditions I have stated above are quite big "if"s, so you need to consider whether it's a good fit for you.

Mapping Collections in Automapper

I'm using Automapper to clone an object. I have a class which contains collections that I want handled in a non-standard way, which I'll explain below (I've stripped out a bunch of code to highlight the specific issue):
public class CommunityModel
{
private readonly IUIManaer _uiMgr;
private readonly IMapper _mapper;
private ValidatedCollection<CommunityUserModel, string> _users;
private int _communityIndex = -1;
public CommunityModel( IUIManager uiMgr, IMapper mapper, IElementManager<CommunityUserModel, string> userMgr )
{
_uiMgr = uiMgr ?? throw new NullReferenceException( nameof(uiMgr) );
_mapper = mapper ?? throw new NullReferenceException( nameof(mapper) );
Users = new ValidatedCollection<CommunityUserModel, string>( userMgr );
}
public int CommunityIndex
{
get => _communityIndex;
set
{
if (value < -1) value = -1;
Set(ref _communityIndex, value);
IsNew = value < 0;
}
}
public ValidatedCollection<CommunityModel, string> Collection { get; set; }
public ValidatedCollection<CommunityUserModel, string> Users
{
get => _users;
set
{
ChangeTracker.RegisterCollection(value);
SetAndValidate( ref _users, value );
}
}
}
ValidatedCollection<> is an extension of WPF's ObservableCollection. The CommunityIndex property uniquely identifies a CommunityModel instance. This allows me to use the Automapper.Collection extensions via the EqualityComparison() extension method.
I don't want Automapper to initialize the Users collection, because it gets initialized in the class constructor. But I want the elements of the User collection to be cloned from the source Users collection to the destination Users collection.
The Collection collection contains a list of CommunityModel objects, including the instance which has the Collection property (i.e., Collection is a set of sibling CommunityModel instances). I'd like Automapper to initialize the collection, and, ultimately, copy all of the CommunityModel siblings other than the source object to that collection, and then add the destination object to the collection (i.e., in the end, Collection will be a set of sibling CommunityModel objects, with the source CommunityModel replaced by the destination CommunityModel).
My map is currently defined as follows:
CreateMap<CommunityModel, CommunityModel>()
.ForMember(dest=>dest.Users, opt=>opt.Ignore())
.ForMember(dest=>dest.Collection, opt=>opt.Ignore() )
.EqualityComparison((x,y)=> x.CommunityIndex == y.CommunityIndex);
CreateMap<CommunityUserModel, CommunityUserModel>()
.EqualityComparison((x,y) => x.UserIndex == y.UserIndex);
If I don't Ignore() the Users collection, Automapper will initialize the collection, which overrides the required configuration of Users set in the constructor and causes problems elsewhere in my app. But if Ignore() the Users collection, its elements are never cloned from source to destination. What I want to do is not have Automapper initialize Users, but still clone the contents.
If I don't ignore the Collection collection, I get an infinite loop and stack overflow, I believe because cloning Collection's elements involves creating an instance of the CommunityModel which owns the Collection property, which should be a reference to the destination object being created, but instead leads to the creation of another identical destination object. What I'd like to do is have Automapper initialize the Collection collection, but >>not<< clone the source elements, which I guess I'd have to do later in an AfterMap() call.
I realize this is somewhat arcane, but the overall design of my project results in these requirements.
What is the best way of doing this within Automapper? Should I look into creating a custom value resolver, even though I'm cloning an object, so the property names are identical between source and destination?

I'm going to describe how I resolved my issue, but I'm not going to mark it as an answer, because I'm not familiar enough with Automapper to know if what I think it's doing is what it's actually doing.
What appears to be happening, when Automapper maps collections -- even with the Automapper.Collections library installed and activated -- is that collections are deemed to be "different", even if the types of the elements in the source and destination collections can be automatically mapped, if the source and destination collection types are different.
For example, if the source Community object has a List<> of CommunityUsers:
public class Community
{
public string Name { get; set; }
public string SiteUrl { get; set; }
public string LoginUrl { get; set; }
public List<CommunityUser> Users { get; set; }
}
public class CommunityUser
{
public string UserID { get; set; }
public string VaultKeyName { get; set; }
}
and you want to map it to destination objects like this:
public class CommunityModel
{
// omitting constructor, which contains logic to
// initialize the Users property based on constructor arguments
public string Name {get;set;}
public string LoginUrl {get;set;}
public string SiteUrl {get;set;}
public ValidatedCollection<CommunityUserModel, string> Users
{ get; set;}
}
public class CommunityUserModel
{
public string UserID {get; set;}
public string VaultKeyName {get;set;}
}
even though all the property names are "recognizable" by Automapper, and the two Users collections are both IEnumerable, the fact that the two collections are of different types apparently causes Automapper to treat the collections as "different".
And that apparently means the add/delete/copy logic of Automapper.Collections doesn't get used, even if present. Instead, Automapper tries to create an instance of (in this case) ValidatedCollection, populate it from the source object collection, and then assign it to destination object collection.
That's fine if ValidatedCollection<> doesn't have required contructor arguments. But it'll fail if it does. Which is what happened in my case.
My workaround was to do this in the Mapper definition:
CreateMap<Community, CommunityModel>()
.ForMember(dest=>dest.Users, opt=> opt.Ignore())
.AfterMap( ( src, dest, rc ) =>
{
foreach( var srcUser in src.Users )
{
dest.Users.Add(rc.Mapper.Map<CommunityUserModel>(srcUser));
}
} );
This keeps Automapper from doing anything with the destination Users property (which is initialized in the CommunityModel constructor), and maps over the source User objects after the "automatic" mapping is done.

Entity Framework: delete list of child objects from parent

I basically have a pretty simple setup of a one-to-many parent-child relationships. E.g.:
public class Parent
{
public Guid Id { get; set; }
public virtual ICollection<Child> Children { get; set; }
}
public class Child
{
public Guid Id { get; set; }
public Guid ParentId { get; set; }
public virtual Parent Parent { get; set; }
}
This is configured using FluentValidation as:
public ChildConfiguration()
{
ToTable("children");
HasKey(c => c.Id);
Property(c => c.Id).HasColumnName("id");
HasRequired(c => c.Parent).WithMany(p => p.Children);
}
I now have an API that provides the list of children for a parent, but I don't want to add the list of children to the existing list every time. Instead I want to delete all existing children to replace them with the list of new ones.
I tried doing so by fetching the parent, clearing the current list of children and adding the new ones.
var parent = await _repository.GetById(parentId);
parent.Children.Clear();
foreach (var child in children)
{
parent.Children.Add(child);
}
This does not work, as it throws an exception:
The operation failed: The relationship could not be changed because one or more of the foreign-key properties is non-nullable. When a change is made to a relationship, the related foreign-key property is set to a null value. If the foreign-key does not support null values, a new relationship must be defined, the foreign-key property must be assigned another non-null value, or the unrelated object must be deleted.
As I understood this is because it's trying to remove the link between the two entities rather than actually deleting it from the database.
A solution is to directly access the Children in the DbContext and do a RemoveRange as:
DbContext.Children.RemoveRange(children);
But I don't want to expose my Children entity directly on the context. Instead I always want to go through the Parent. So I could create a method in the repository like:
var parent = DbContext.Parents.Find(parentId);
foreach (var child in parent.Children.ToList())
{
parent.Remove(child);
}
But this is throwing the same exception and I don't understand why as I think I am explicitly telling EF to remove the child from the parent.
Is there a way to remove the children from the DB without accessing them directly on the DbContext?
Thanks!

Guid is a struct and struct is non-nullable type like primitive types.
eg : int, double, bool, etc...
You will be able to delete the children of a given parent with your following code
var parent = await _repository.GetById(parentId);
parent.Children.Clear();
foreach (var child in children)
{
parent.Children.Add(child);
}
by declaring your Guid ParentId property in you Child class Nullable like this Nullable<Guid>.

Nested parent child lists where order by doesn't work

I have the following code:
public class Navigation
{
public Navigation()
{
SubNavigation = new List<Navigation>();
}
public int Order { get; set; }
public string Text { get; set; }
public string RouteName { get; set; }
public IList<Navigation> SubNavigation { get; set; }
}
I then have:
IList<Navigation> list = new List<Navigation>();
I populate the list with some data. Not all items have a sub navigation. Currently the navigation only goes one level deep.
Now I would like to sort both the navigation and the sub-navigation for each item by order. I have tried all kinds of approaches but no matter what I tried I could not get the sub-navigation to sort without re-creating the object. The below code works:
IList<Navigation> result = list.OrderBy(l => l.Order)
.Select(n => new Navigation
{
Order = n.Order,
Text = n.Text,
RouteName = n.RouteName,
SubNavigation = n.SubNavigation.OrderBy(s => s.Order).ToList()
}).ToList();
I am not in love with this approach and my question is if there is any cleaner/better way of doing this using LINQ and the method syntax?

You could add a new property on your object:
public IList<Navigation> OrderedSubNavigation
{
get
{
return SubNavigation.OrderBy(s => s.Order).ToList();
}
}
Then when you want the ordered one you just use that.

I have tried all kinds of approaches but no matter what I tried I could not get the sub-navigation to sort without re-creating the object.
Well no, you wouldn't be able to cleanly - because getting the subnavigation to be in a particular order requires modifying the existing object, and LINQ's not built for that. LINQ's built for queries, which shouldn't mutate the data they work on.
One option would be to only sort the subnavigation when you need to - live with the fact that it's unordered within a Navigation, and then when you actually need the subnavigation items (e.g. for display) you can order at that point. Aside from anything else, this will make it more efficient if you end up not displaying the subnavigation items.