We Keep Coding

Traversing a graph of unknown object types and mutating some object properties

The specific requirement is to replace some values in some MVC model properties if a user doesn't have specific permission.
The solution should be universal to apply to any graph of any model, and also reasonably efficient because it will be used to mask values in large lists of objects.
The assumptions are:
the graph consists of custom objects of unknown type (it's only known these will be C# classes)
all the objects of interest have public properties and only the public properties should be inspected (possibly, using custom [SensitiveData] attribute as a filter, mostly for performance reasons to ignore most of the properties)
the objects may have other child objects that need to be traversed for [SensitiveData] properties
the objects may have properties of different types of collections that need to be traversed (IEnumerables, ILists, generic IEnumerables and ILists with custom objects)
the objects may also contain collections of .NET core objects, such as IEnumerable<int>, IEnumerable<DateTime> which is of no interest to me and shouldn't be traversed
I don't want to traverse properties of .NET core objects (no need to inspect Date property of DateTime)
I don't want to iterate through .NET core objects (no need to inspect every character of string, which is IEnumerable)
in most cases it will be a tree and not a graph. Still, it would be safer to implement it for graphs with protection against getting in endless loops.
It all seems to be pretty logic and common requirements, so I thought there should be some generic, well-tested solution I could adjust to my case, just passing a callback func and maybe some filter to define properties of interest before even starting traversal.
However, so far, everything I find is limited to single Node-like type or the implementation doesn't allow to mutate properties of my choice or it goes into deep recursions with reflections and without any performance considerations.
I could implement it myself but it might turn out to be reinventing the wheel with messy recursions and reflections. Isn't there anything already existing and well-known that "just works"?
Also, I've heard that reflection SetValue and GetValue methods are slow and I should better cache setters and getters as delegates and reuse them whenever I encounter the same type again. And I will encounter the same types again because it's a ASP.NET Core web application. So it could be possible to gain noticeable performance boost over naive reflection solutions if I cache every setter/getter of interest for future reuse.

It took some struggling but with awesome graph traversal example from Eric Lippert and FastMember library I have something that works:
[AttributeUsage(AttributeTargets.Property, AllowMultiple = false, Inherited = true)]
public class SensitiveDataAttribute : Attribute
{
}
public abstract class PocoGraphPropertyWalker
{
private enum TypeKind
{
Primitive,
IterablePrimitive,
Poco,
IterablePoco
}
private class TypeAccessDescriptor
{
public TypeAccessor accessor;
public List<PropertyInfo> primitives;
public List<PropertyInfo> iterables;
public List<PropertyInfo> singles;
}
private static ConcurrentDictionary<Type, TypeAccessDescriptor> _accessorCache =
new ConcurrentDictionary<Type, TypeAccessDescriptor>();
public IEnumerable<object> TraversePocoList(IEnumerable<object> pocos)
{
if (pocos == null)
return null;
foreach (var poco in pocos)
TraversePoco(poco);
return pocos;
}
public object TraversePoco(object poco)
{
var unwound = Traversal(poco, ChildrenSelector).ToList();
foreach(var unw in unwound)
VisitPoco(unw);
return poco;
}
public object VisitPoco(object poco)
{
if (poco == null)
return poco;
var t = poco.GetType();
// the registry ignores types that are not POCOs
var typeDesc = TryGetOrRegisterForType(t);
if (typeDesc == null)
return poco;
// do not attempt to parse Keys and Values as primitives,
// even if they were specified as such
if (IsKeyValuePair(t))
return poco;
foreach (var prop in typeDesc.primitives)
{
var oldValue = typeDesc.accessor[poco, prop.Name];
var newValue = VisitProperty(poco, oldValue, prop);
typeDesc.accessor[poco, prop.Name] = newValue;
}
return poco;
}
protected virtual object VisitProperty(object model,
object currentValue, PropertyInfo prop)
{
return currentValue;
}
private IEnumerable<object> Traversal(
object item,
Func<object, IEnumerable<object>> children)
{
var seen = new HashSet<object>();
var stack = new Stack<object>();
seen.Add(item);
stack.Push(item);
yield return item;
while (stack.Count > 0)
{
object current = stack.Pop();
foreach (object newItem in children(current))
{
// protect against cyclic refs
if (!seen.Contains(newItem))
{
seen.Add(newItem);
stack.Push(newItem);
yield return newItem;
}
}
}
}
private IEnumerable<object> ChildrenSelector(object poco)
{
if (poco == null)
yield break;
var t = poco.GetType();
// the registry ignores types that are not POCOs
var typeDesc = TryGetOrRegisterForType(t);
if (typeDesc == null)
yield break;
// special hack for KeyValuePair - FastMember fails to access its Key and Value
// maybe because it's a struct, not class?
// and now we have prop accessors stored in singles / primitives
// so we extract it manually
if (IsKeyValuePair(t))
{
// reverting to good old slow reflection
var k = t.GetProperty("Key").GetValue(poco, null);
var v = t.GetProperty("Value").GetValue(poco, null);
if (k != null)
{
foreach (var yp in YieldIfPoco(k))
yield return yp;
}
if (v != null)
{
foreach(var yp in YieldIfPoco(v))
yield return yp;
}
yield break;
}
// registration method should have registered correct singles
foreach (var single in typeDesc.singles)
{
yield return typeDesc.accessor[poco, single.Name];
}
// registration method should have registered correct IEnumerables
// to skip strings as enums and primitives as enums
foreach (var iterable in typeDesc.iterables)
{
if (!(typeDesc.accessor[poco, iterable.Name] is IEnumerable iterVals))
continue;
foreach (var iterval in iterVals)
yield return iterval;
}
}
private IEnumerable<object> YieldIfPoco(object v)
{
var myKind = GetKindOfType(v.GetType());
if (myKind == TypeKind.Poco)
{
foreach (var d in YieldDeeper(v))
yield return d;
}
else if (myKind == TypeKind.IterablePoco && v is IEnumerable iterVals)
{
foreach (var i in iterVals)
foreach (var d in YieldDeeper(i))
yield return d;
}
}
private IEnumerable<object> YieldDeeper(object o)
{
yield return o;
// going slightly recursive here - might have IEnumerable<IEnumerable<IEnumerable<POCO>>>...
var chs = Traversal(o, ChildrenSelector);
foreach (var c in chs)
yield return c;
}
private TypeAccessDescriptor TryGetOrRegisterForType(Type t)
{
if (!_accessorCache.TryGetValue(t, out var typeAccessorsDescriptor))
{
// blacklist - cannot process dictionary KeyValues
if (IsBlacklisted(t))
return null;
// check if I myself am a real Poco before registering my properties
var myKind = GetKindOfType(t);
if (myKind != TypeKind.Poco)
return null;
var properties = t.GetProperties(BindingFlags.Public | BindingFlags.Instance);
var accessor = TypeAccessor.Create(t);
var primitiveProps = new List<PropertyInfo>();
var singlePocos = new List<PropertyInfo>();
var iterablePocos = new List<PropertyInfo>();
// now sort all props in subtypes:
// 1) a primitive value or nullable primitive or string
// 2) an iterable with primitives (including strings and nullable primitives)
// 3) a subpoco
// 4) an iterable with subpocos
// for our purposes, 1 and 2 are the same - just properties,
// not needing traversion
// ignoring non-generic IEnumerable - can't know its inner types
// and it is not expected to be used in our POCOs anyway
foreach (var prop in properties)
{
var pt = prop.PropertyType;
var propKind = GetKindOfType(pt);
// 1) and 2)
if (propKind == TypeKind.Primitive || propKind == TypeKind.IterablePrimitive)
primitiveProps.Add(prop);
else
if (propKind == TypeKind.IterablePoco)
iterablePocos.Add(prop); //4)
else
singlePocos.Add(prop); // 3)
}
typeAccessorsDescriptor = new TypeAccessDescriptor {
accessor = accessor,
primitives = primitiveProps,
singles = singlePocos,
iterables = iterablePocos
};
if (!_accessorCache.TryAdd(t, typeAccessorsDescriptor))
{
// if failed add, a parallel process added it, just get it back
if (!_accessorCache.TryGetValue(t, out typeAccessorsDescriptor))
throw new Exception("Failed to get a type descriptor that should exist");
}
}
return typeAccessorsDescriptor;
}
private static TypeKind GetKindOfType(Type type)
{
// 1) a primitive value or nullable primitive or string
// 2) an iterable with primitives (including strings and nullable primitives)
// 3) a subpoco
// 4) an iterable with subpocos
// ignoring non-generic IEnumerable - can't know its inner types
// and it is not expected to be used in our POCOs anyway
// 1)
if (IsSimpleType(type))
return TypeKind.Primitive;
var ienumerableInterfaces = type.GetInterfaces()
.Where(x => x.IsGenericType && x.GetGenericTypeDefinition() ==
typeof(IEnumerable<>)).ToList();
// add itself, if the property is defined as IEnumerable<x>
if (type.IsGenericType && type.GetGenericTypeDefinition() ==
typeof(IEnumerable<>))
ienumerableInterfaces.Add(type);
if (ienumerableInterfaces.Any(x =>
IsSimpleType(x.GenericTypeArguments[0])))
return TypeKind.IterablePrimitive;
if (ienumerableInterfaces.Count() != 0)
// 4) - it was enumerable, but not primitive - maybe POCOs
return TypeKind.IterablePoco;
return TypeKind.Poco;
}
private static bool IsBlacklisted(Type type)
{
return false;
}
public static bool IsKeyValuePair(Type type)
{
return type.IsGenericType &&
type.GetGenericTypeDefinition() == typeof(KeyValuePair<,>);
}
public static bool IsSimpleType(Type type)
{
return
type.IsPrimitive ||
new Type[] {
typeof(string),
typeof(decimal),
typeof(DateTime),
typeof(DateTimeOffset),
typeof(TimeSpan),
typeof(Guid)
}.Contains(type) ||
type.IsEnum ||
Convert.GetTypeCode(type) != TypeCode.Object ||
(type.IsGenericType && type.GetGenericTypeDefinition() == typeof(Nullable<>) && IsSimpleType(type.GetGenericArguments()[0]))
;
}
}
public class ProjectSpecificDataFilter : PocoGraphPropertyWalker
{
const string MASK = "******";
protected override object VisitProperty(object model,
object currentValue, PropertyInfo prop)
{
if (prop.GetCustomAttributes<SensitiveDataAttribute>().FirstOrDefault() == null)
return currentValue;
if (currentValue == null || (currentValue is string &&
string.IsNullOrWhiteSpace((string)currentValue)))
return currentValue;
return MASK;
}
}
For testing:
enum MyEnum
{
One = 1,
Two = 2
}
class A
{
[SensitiveData]
public string S { get; set; }
public int I { get; set; }
public int? I2 { get; set; }
public MyEnum Enm { get; set; }
public MyEnum? Enm1 { get; set; }
public List<MyEnum> Enm2 { get; set; }
public List<int> IL1 { get; set; }
public int?[] IL2 { get; set; }
public decimal Dc { get; set; }
public decimal? Dc1 { get; set; }
public IEnumerable<decimal> Dc3 { get; set; }
public IEnumerable<decimal?> Dc4 { get; set; }
public IList<decimal> Dc5 { get; set; }
public DateTime D { get; set; }
public DateTime? D2 { get; set; }
public B Child { get; set; }
public B[] Children { get; set; }
public List<B> Children2 { get; set; }
public IEnumerable<B> Children3 { get; set; }
public IDictionary<int, int?> PrimDict { get; set; }
public Dictionary<int, B> PocoDict { get; set; }
public IDictionary<B, int?> PocoKeyDict { get; set; }
public Dictionary<int, IEnumerable<B>> PocoDeepDict { get; set; }
}
class B
{
[SensitiveData]
public string S { get; set; }
public int I { get; set; }
public int? I2 { get; set; }
public DateTime D { get; set; }
public DateTime? D2 { get; set; }
public A Parent { get; set; }
}
class Program
{
static A root;
static void Main(string[] args)
{
root = new A
{
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy",
Child = new B
{
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy"
},
Children = new B[] {
new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" },
new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" },
},
Children2 = new List<B> {
new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" },
new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" },
},
PrimDict = new Dictionary<int, int?> {
{ 1, 2 },
{ 3, 4 }
},
PocoDict = new Dictionary<int, B> {
{ 1, new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" } },
{ 3, new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" } }
},
PocoKeyDict = new Dictionary<B, int?> {
{ new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" }, 1 },
{ new B {
D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" }, 3 }
},
PocoDeepDict = new Dictionary<int, IEnumerable<B>>
{
{ 1, new [] { new B {D = DateTime.Now,
D2 = DateTime.Now,
I = 10,
I2 = 20,
S = "stringy" } } }
}
};
// add cyclic ref for test
root.Child.Parent = root;
var f = new VtuaSensitiveDataFilter();
var r = f.TraversePoco(root);
}
}
It replaces marked strings, no matter how deep inside POCOs they are.
Also I can use the walker for every other property access / mutation case I can imagine.
Still not sure if I reinvented a wheel here...

How to correctly use Expression Trees on properties (without reflection)

Objective: process an object and if the object implements an expected type, I want to change a specific property value (this part is working fine), and I also would like to apply the same logic to all property lists (that I explicit point) that are of the same expected type.
I have the following code:
public abstract class BaseObject
{
public int Id { get; set; }
}
public class Father : BaseObject
{
public DateTime CreatedOn { get; set; }
public string Name { get; set; }
public IEnumerable<ChildA> Children1 { get; set; }
public IEnumerable<ChildB> Children2 { get; set; }
public IEnumerable<ChildA> Children3 { get; set; }
public IEnumerable<ChildB> Children4 { get; set; }
}
public class ChildA : BaseObject
{
public int Val1 { get; set; }
}
public class ChildB : BaseObject
{
public string Name { get; set; }
public int Total { get; set; }
}
I want to process an object by applying some changes on a specific property on the target object and on all property children that I explicit say:
public void Start()
{
var listA = new List<ChildA> { new ChildA { Id = 1, Val1 = 1 }, new ChildA { Id = 2, Val1 = 2 } };
var listB = new List<ChildB> { new ChildB { Id = 1, Name = "1", Total = 1 } };
var obj = new Father { Id = 1, CreatedOn = DateTime.Now, Name = "F1", ChildrenA = listA, ChildrenB = listB };
// I explicit tell to process only 2 of the 4 lists....
ProcessObj(obj, x => new object[] { x.Children1, x.Children2 });
}
I was able to write this function:
public void ProcessObj<T>(T obj, Expression<Func<T, object[]>> includes = null)
{
var objBaseObject = obj as BaseObject;
if (objBaseObject == null) return;
// Here I change the ID - add 100 just as an example....
objBaseObject.Id = objBaseObject.Id + 100;
if (includes == null) return;
var array = includes.Body as NewArrayExpression;
if (array == null) return;
var exps = ((IEnumerable<object>)array.Expressions).ToArray();
for (var i = 0; i < exps.Count(); i++)
{
var name = ((MemberExpression)exps[i]).Member.Name;
var childProperty = obj.GetType().GetProperties(
BindingFlags.Public | BindingFlags.Instance
).FirstOrDefault(prop => prop.Name == name);
if (childProperty == null) continue;
// NOT correct because I think I am getting a copy of the object
// and not pointing to the object in memory (by reference)
var childList = childProperty.GetValue(obj);
// TODO: loop on the list and apply the same logic as the father....
// change the ID field....
}
}
In this prototype I started writing reflection, but I really would like to avoid it if possible....
How can I do this???

Maybe I'm missing something, but it seems like you're complicating the problem by using expression trees. Can you just not use a regular Action and Func delegates to do this? Why do they need to be expression trees? Here's an example just using delegates:
public void ProcessObj<T>(T obj, Func<T, IEnumerable<object>> includes) {
var objBaseObject = obj as BaseObject;
if (objBaseObject == null) return;
// Create a reusable action to use on both the parent and the children
Action<BaseObject> action = x => x.Id += 100;
// Run the action against the root object
action(objBaseObject);
// Get the includes by just invoking the delegate. No need for trees.
var includes = includes(obj);
// Loop over each item in each collection. If the types then invoke the same action that we used on the root.
foreach(IEnumerable<object> include in includes)
{
foreach(object item in include)
{
var childBaseObject = item as BaseObject;
if(childBaseObject != null)
{
action(childBaseObject);
}
}
}
}
Useable just like before:
ProcessObj(obj, x => new object[] { x.Children1, x.Children2 });
No expression trees and no reflection, just regular delegate lambdas.
Hope that helps

CRUD over two different collections of POCOs

I have an interesting question, one I'm having difficulty searching for an answer on.
I have two IEnumerable collections of objects. The underlying objects are completely separate, BUT I can identify a shared key that should match. The collections are important, in that my "left" object is the "system of record", and the "right" object is representing a system I need to ensure matches the system of record.
Once they are matched, I need to perform CRUD operations on one side to bring the right side in line with the left side. For example, it would create a new item on the right side if one didn't exist, or update values, or delete if the item was missing on the left, but not the right.
The catch is, I have hundreds of these collections to match up, and the actual CRUD code is different.
I'd like to introduce some shared code where I can pass in both collections, the collection types (as probably generics), some kind of comparer, and some delegates of what operation to perform for CRUD.
If this code actually existed, it may look something like this
class Stuff
{
string Id {get; set;}
string Name {get; set;}
}
class Junk
{
string Id {get; set;}
string ShortName {get; set;}
}
IEnumerable<Stuff> myStuff = GetStuff();
IEnumerable<Junk> myJunk = GetJunk();
CrudComparer cc = new CrudComparer<Stuff, Junk>(myStuff, myJunk);
cc.Comparer = (leftObject, rightObject) => {
leftObject.Name == rightObject.Name
}
cc.CreateOperation = (newObject, rightCollection) => {
Junk j = new Junk();
j.Shortname = newObject.Name;
rightCollection.Add(j);
}
cc.UpdateOperation = (leftObject, rightObject) => {
rightObject.Shortname = leftObject.Name;
}
cc.DeleteOperation = (rightCollection, rightObject) => {
rightCollection.Remove(rightObject);
}
cc.Compare();
Has anyone ever seen code that does something like this? I'd hate to reinvent the wheel if I can grab something already done.
Thanks for any help!
--Michael

I got to thinking more about this, and realized what I knew about delgates and generics should be sufficient to solve this problem, so I got in LinqPad and had some fun. I haven't written any unit tests around this yet, so use at your own risk, but hopefully if you want to use this you understand the underlying concepts.
class Blah
{
public int ID { get; set; }
public string BlahName { get; set;}
}
class Bleh
{
public int ID { get; set; }
public string BlehName { get; set;}
}
class CrudComparer<TLeft, TRight>
{
private readonly ICollection<TLeft> _leftCollection;
private readonly ICollection<TRight> _rightCollection;
private readonly Comparer _compareOperation;
private readonly CreateOperation _createOperation;
private readonly UpdateOperation _updateOperation;
private readonly DeleteOperation _deleteOperation;
public delegate bool Comparer(TLeft leftItem, TRight rightItem);
public delegate void CreateOperation(TLeft leftItem, ICollection<TRight> rightCollection);
public delegate void UpdateOperation(TLeft leftItem, TRight rightItem);
public delegate void DeleteOperation(TRight rightItem, ICollection<TRight> rightCollection);
public CrudComparer(ICollection<TLeft> leftCollection, ICollection<TRight> rightCollection, Comparer compareOperation, CreateOperation createOperation, UpdateOperation updateOperation, DeleteOperation deleteOperation)
{
_leftCollection = leftCollection;
_rightCollection = rightCollection;
_compareOperation = compareOperation;
_createOperation = createOperation;
_updateOperation = updateOperation;
_deleteOperation = deleteOperation;
}
public void Compare()
{
foreach (TLeft leftItem in _leftCollection)
{
bool foundItem = false;
foreach (TRight rightItem in _rightCollection)
{
if (_compareOperation(leftItem, rightItem))
{
//these equal
foundItem = true;
}
}
if (foundItem == false)
{
_createOperation(leftItem, _rightCollection);
}
}
List<TRight> itemsToDelete = new List<TRight>();
foreach (TRight rightItem in _rightCollection)
{
bool foundItem = false;
foreach (TLeft leftItem in _leftCollection)
{
if (_compareOperation(leftItem, rightItem))
{
foundItem = true;
_updateOperation(leftItem, rightItem);
break;
}
}
if (!foundItem)
{
itemsToDelete.Add(rightItem);
}
}
foreach (TRight itemToDelete in itemsToDelete)
{
_deleteOperation(itemToDelete, _rightCollection);
}
}
}
void Main()
{
List<Blah> blahItems = new List<Blah>();
blahItems.Add(new Blah() { ID = 1, BlahName = "Blah" });
blahItems.Add(new Blah() { ID = 2, BlahName = "ABC" });
blahItems.Add(new Blah() { ID = 34, BlahName = "XYZ" });
blahItems.Add(new Blah() { ID = 6442, BlahName = "123" });
List<Bleh> blehItems = new List<Bleh>();
blehItems.Add(new Bleh() { ID = 2, BlehName = "12345"});
blehItems.Add(new Bleh() { ID = 6, BlehName = "43232"});
blehItems.Add(new Bleh() { ID = 77, BlehName = "BlahBlah"});
blehItems.Add(new Bleh() { ID = 2334, BlehName = "ZYX"});
CrudComparer<Blah, Bleh>.Comparer compareOperation = (leftObject, rightObject) =>
{
return leftObject.ID == rightObject.ID;
};
CrudComparer<Blah, Bleh>.CreateOperation createOperation = (leftObject, rightCollection) =>
{
rightCollection.Add(new Bleh() { ID = leftObject.ID });
};
CrudComparer<Blah, Bleh>.UpdateOperation updateOperation = (leftObject, rightObject) =>
{
rightObject.BlehName = leftObject.BlahName;
};
CrudComparer<Blah, Bleh>.DeleteOperation deleteOperation = (rightObject, rightCollection) =>
{
rightCollection.Remove(rightObject);
};
CrudComparer<Blah, Bleh> cc = new CrudComparer<Blah, Bleh>(blahItems, blehItems, compareOperation, createOperation, updateOperation, deleteOperation);
cc.Compare();
}

Get Values From Complex Class Using Reflection

I have a class, which is created and populated from an xml string, I've simplified it for example purposes:
[XmlRoot("Person")]
public sealed class Person
{
[XmlElement("Name")]
public string Name { get; set; }
[XmlElement("Location")]
public string Location { get; set; }
[XmlElement("Emails", Type = typeof(PersonEmails)]
public PersonEmails Emails { get; set; }
}
public class PersonEmails
{
[XmlElement("Email", Type = typeof(PersonEmail))]
public PersonEmail[] Emails { get; set; }
}
public class PersonEmail
{
[XmlAttribute("Type")]
public string Type { get; set; }
[XmlText]
public string Value { get; set; }
}
To extract the information, I'm trying to load them into another class, which is simply:
public class TransferObject
{
public string Name { get; set; }
public ObjectField[] Fields { get; set; }
}
public class ObjectField
{
public string Name { get; set; }
public string Value { get; set; }
}
I'm only populating "Fields" from the other object, which would simply be (Name = "Location", Value = "London"), but for Emails, (Name = "Email"+Type, Value = jeff#here.com)
Currently I can populate all the other fields, but I'm stuck with Emails, and knowing how to dig deep enough to be able to use reflection (or not) to get the information I need. Currently I'm using:
Person person = Person.FromXmlString(xmlString);
List<ObjectField> fields = new List<ObjectField>();
foreach (PropertyInfo pinfo in person.getType().GetProperties()
{
fields.Add(new ObjectField { Name = pinfo.Name, Value = pinfo.getValue(person, null).ToString();
}
How can I expand on the above to add all my emails to the list?

You are trying to type cast a complex values type to string value so you lost the data. Instead use following code:
class Program
{
static void Main(string[] args)
{
Person person = new Person();
person.Name = "Person One";
person.Location = "India";
person.Emails = new PersonEmails();
person.Phones = new PersonPhones();
person.Emails.Emails = new PersonEmail[] { new PersonEmail() { Type = "Official", Value = "xyz#official.com" }, new PersonEmail() { Type = "Personal", Value = "xyz#personal.com" } };
person.Phones.Phones = new PersonPhone[] { new PersonPhone() { Type = "Official", Value = "789-456-1230" }, new PersonPhone() { Type = "Personal", Value = "123-456-7890" } };
List<ObjectField> fields = new List<ObjectField>();
fields = GetPropertyValues(person);
}
static List<ObjectField> GetPropertyValues(object obj)
{
List<ObjectField> propList = new List<ObjectField>();
foreach (PropertyInfo pinfo in obj.GetType().GetProperties())
{
var value = pinfo.GetValue(obj, null);
if (pinfo.PropertyType.IsArray)
{
var arr = value as object[];
for (var i = 0; i < arr.Length; i++)
{
if (arr[i].GetType().IsPrimitive)
{
propList.Add(new ObjectField() { Name = pinfo.Name + i.ToString(), Value = arr[i].ToString() });
}
else
{
var lst = GetPropertyValues(arr[i]);
if (lst != null && lst.Count > 0)
propList.AddRange(lst);
}
}
}
else
{
if (pinfo.PropertyType.IsPrimitive || value.GetType() == typeof(string))
{
propList.Add(new ObjectField() { Name = pinfo.Name, Value = value.ToString() });
}
else
{
var lst = GetPropertyValues(value);
if (lst != null && lst.Count > 0)
propList.AddRange(lst);
}
}
}
return propList;
}
}

Check this snippet out:
if(pinfo.PropertyType.IsArray)
{
// Grab the actual instance of the array.
// We'll have to use it in a few spots.
var array = pinfo.GetValue(personObject);
// Get the length of the array and build an indexArray.
int length = (int)pinfo.PropertyType.GetProperty("Length").GetValue(array);
// Get the "GetValue" method so we can extact the array values
var getValue = findGetValue(pinfo.PropertyType);
// Cycle through each index and use our "getValue" to fetch the value from the array.
for(int i=0; i<length; i++)
fields.Add(new ObjectField { Name = pinfo.Name, Value = getValue.Invoke(array, new object[]{i}).ToString();
}
// Looks for the "GetValue(int index)" MethodInfo.
private static System.Reflection.MethodInfo findGetValue(Type t)
{
return (from mi in t.GetMethods()
where mi.Name == "GetValue"
let parms = mi.GetParameters()
where parms.Length == 1
from p in parms
where p.ParameterType == typeof(int)
select mi).First();
}
You can definately do it with Reflection... You can take advantage of the fact that a Type can tell you if it's an array or not (IsArray)... and then take advantage of the fact that an Array has a method GetValue(int index) that will give you a value back.
Per your comment
Because Emails is a property within a different class, recursion should be used. However the trick is knowing when to go to the next level. Really that is up to you, but
if it were me, I would use some sort of Attribute:
static void fetchProperties(Object instance, List<ObjectField> fields)
{
foreach(var pinfo in instance.GetType().GetProperties())
{
if(pinfo.PropertyType.IsArray)
{
... // Code described above
}
else if(pinfo.PropertyType.GetCustomAttributes(typeof(SomeAttribute), false).Any())
// Go the next level
fetchProperties(pinfo.GetValue(instance), fields);
else
{
... // Do normal code
}
}
}

Schema dependent class to schemaless document using MongoDb and C#

Let us suppose we have a document to store our client which has fixed and extra fields.
So here goes our sample class for the client:
public class Client
{
public string Name{ get; set; }
public string Address{ get; set; }
public List<ExtraField> ExtraFields{ get; set; } //these fields are extra ones
}
In extra field class we have something like this:
public class ExtraField
{
public string Key{ get; set; }
public string Type { get; set; }
public string Value { get; set; }
}
If I use standard driver's behaviour for serialization I would get smth like this:
{{Name:VName, Address:VAddress, ExtraFields:[{Key:VKey,Type:VType,
Value:VValue},...]}, document2,...,documentn}
While I would like to have something like this:
{{Name:VName, Address:VAddress, VKey:VValue,...}, document2,...,documentn}
This would improve the search performance and is generally the point of document orientation.
How can I customize the serialization to such a way?

Here is the way I solved it (it works fine) and solved the issue.
using System;
using System.Collections.Generic;
using System.Linq; using System.Text;
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Bson.Serialization.Serializers;
namespace TestDataGeneration {
public class FieldsWrapper : IBsonSerializable
{
public List<DataFieldValue> DataFieldValues { get; set; }
public object Deserialize(MongoDB.Bson.IO.BsonReader bsonReader, Type nominalType, IBsonSerializationOptions options)
{
if (nominalType != typeof(FieldsWrapper)) throw new ArgumentException("Cannot deserialize anything but self");
var doc = BsonDocument.ReadFrom(bsonReader);
var list = new List<DataFieldValue>();
foreach (var name in doc.Names)
{
var val = doc[name];
if (val.IsString)
list.Add(new DataFieldValue {LocalIdentifier = name, Values = new List<string> {val.AsString}});
else if (val.IsBsonArray)
{
DataFieldValue df = new DataFieldValue {LocalIdentifier = name};
foreach (var elem in val.AsBsonArray)
{
df.Values.Add(elem.AsString);
}
list.Add(df);
}
}
return new FieldsWrapper {DataFieldValues = list};
}
public void Serialize(MongoDB.Bson.IO.BsonWriter bsonWriter, Type nominalType, IBsonSerializationOptions options)
{
if (nominalType != typeof (FieldsWrapper))
throw new ArgumentException("Cannot serialize anything but self");
bsonWriter.WriteStartDocument();
foreach (var dataFieldValue in DataFieldValues)
{
bsonWriter.WriteName(dataFieldValue.LocalIdentifier);
if (dataFieldValue.Values.Count != 1)
{
var list = new string[dataFieldValue.Values.Count];
for (int i = 0; i < dataFieldValue.Values.Count; i++)
list[i] = dataFieldValue.Values[i];
BsonSerializer.Serialize(bsonWriter, list);
}
else
{
BsonSerializer.Serialize(bsonWriter, dataFieldValue.Values[0]);
}
}
bsonWriter.WriteEndDocument();
}
} }

Essentially you just need to implement two methods yourself. First one to serialize an object as you want and second to deserialize an object from db to your Client class back:
1 Seialize client class:
public static BsonValue ToBson(Client client)
{
if (client == null)
return null;
var doc = new BsonDocument();
doc["Name"] = client.Name;
doc["Address"] = client.Address;
foreach (var f in client.ExtraFields)
{
var fieldValue = new BsonDocument();
fieldValue["Type"] = f.Type;
fieldValue["Value"] = f.Value;
doc[f.Key] = fieldValue;
}
return doc;
}
2 Deserialize client object:
public static Client FromBson(BsonValue bson)
{
if (bson == null || !bson.IsBsonDocument)
return null;
var doc = bson.AsBsonDocument;
var client = new Client
{
ExtraFields = new List<ExtraField>(),
Address = doc["Address"].AsString,
Name = doc["Name"].AsString
};
foreach (var name in doc.Names)
{
var val = doc[name];
if (val is BsonDocument)
{
var fieldDoc = val as BsonDocument;
var field = new ExtraField
{
Key = name,
Value = fieldDoc["Value"].AsString,
Type = fieldDoc["Type"].AsString
};
client.ExtraFields.Add(field);
}
}
return client;
}
3 Complete test example:
I've added above two method to your client class.
var server = MongoServer.Create("mongodb://localhost:27020");
var database = server.GetDatabase("SO");
var clients = database.GetCollection<Type>("clients");
var client = new Client() {Id = ObjectId.GenerateNewId().ToString()};
client.Name = "Andrew";
client.Address = "Address";
client.ExtraFields = new List<ExtraField>();
client.ExtraFields.Add(new ExtraField()
{
Key = "key1",
Type = "type1",
Value = "value1"
});
client.ExtraFields.Add(new ExtraField()
{
Key = "key2",
Type = "type2",
Value = "value2"
});
//When inseting/saving use ToBson to serialize client
clients.Insert(Client.ToBson(client));
//When reading back from the database use FromBson method:
var fromDb = Client.FromBson(clients.FindOneAs<BsonDocument>());
4 Data structure in a database:
{
"_id" : ObjectId("4e3a66679c66673e9c1da660"),
"Name" : "Andrew",
"Address" : "Address",
"key1" : {
"Type" : "type1",
"Value" : "value1"
},
"key2" : {
"Type" : "type2",
"Value" : "value2"
}
}
BTW: Take a look into serialization tutorial as well.