We Keep Coding

Linqkit: Prevent circular references and stackoverflow

I'm using LinqKit with EntityFrameWorkCore to create projections with single element projections. But since some Models are nested I'm getting a stackoverflow. I tried to add a condition to the Projection method (bool mapCollusions) to prevent this, but it seems to be ignored. Does anyone have an idea how to prevent these circluar references?
public static Expression<Func<Transport, TransportModel>> GetMainProjection()
{
return transport => new TransportModel()
{
Inmate = transport.Inmate != null ? InmateProjectionsGetProjection(true).Invoke(transport.Inmate) : null,
};
public static Expression<Func<Inmate, InmateModel>> InmateProjectionsGetProjection(bool mapCollusions)
{
return inmate => new InmateModel()
{
Collusions = mapCollusions ? inmate.Collusions.AsQueryable()
.Select(collusion => CollusionProjectionsGetProjection(false).Invoke(collusion))
.ToList() : null
};
}
public static Expression<Func<Collusion, CollusionModel>> CollusionProjectionsGetProjection(bool mapInmate)
{
return collusion => new CollusionModel()
{
Inmate = mapInmate ? InmateProjectionsGetProjection(false).Invoke(collusion.Inmate) : null,
};
}

Try the following realisation:
public static Expression<Func<Transport, TransportModel>> GetMainProjection()
{
return transport => new TransportModel()
{
Inmate = transport.Inmate != null ? InmateProjectionsGetProjection(true).Invoke(transport.Inmate) : null,
};
}
public static Expression<Func<Inmate, InmateModel>> InmateProjectionsGetProjection(bool mapCollusions)
{
if (!mapCollusions)
return inmate => new InmateModel();
return inmate => new InmateModel()
{
Collusions = nmate.Collusions.AsQueryable()
.Select(collusion => CollusionProjectionsGetProjection(false).Invoke(collusion))
.ToList()
};
}
public static Expression<Func<Collusion, CollusionModel>> CollusionProjectionsGetProjection(bool mapInmate)
{
if (!mapInmate)
return collusion => new CollusionModel();
return collusion => new CollusionModel()
{
Inmate = InmateProjectionsGetProjection(false).Invoke(collusion.Inmate),
};
}

The type appears in two structurally incompatible initializations within a single LINQ to Entities query

I'm trying to build something like conditional queries to get only needed data from the underlying database.
Currently I have the following query (which works fine)
var eventData = dbContext.Event.Select(t => new
{
Address = true ? new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
} : new AnonymousEventGetAddress(),
});
If I change it to
var includeAddress = true; // this will normally be passed as param
var eventData = dbContext.Event.Select(t => new
{
Address = includeAddress ? new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
} : new AnonymousEventGetAddress(),
});
I get the following error:
The type 'AnonymousEventGetAddress' appears in two structurally incompatible initializations within a single LINQ to Entities query.
A type can be initialized in two places in the same query, but only if the same properties are set in both places and those properties are set in the same order.
What am I doing wrong here (as of with the true it's working) and how can this be fixed?
I know that changing the else-part to
new AnonymousEventGetAddress
{
AddressLine1 = null,
CityName = null
}
will work. But if I change the order of the properties then, this will also fail.
The class used is defined the following:
public class AnonymousEventGetAddress : BaseAnonymousObject<AnonymousEventGetAddress>
{
public string AddressLine1 { get; set; }
public string CityName { get; set; }
}
whereas BaseAnonymousObject<AnonymousEventGetAddress> is defined:
public abstract class BaseAnonymousObject<TAnonymous>
where TAnonymous : BaseAnonymousObject<TAnonymous>
{
// this is used in case I have to return a list instead of a single anonymous object
public static Expression<Func<IEnumerable<TAnonymous>>> Empty => () => new TAnonymous[]{}.AsEnumerable();
}

I don't know why EF has such requirement, but the important thing is that the requirement exists and we need to take it into account.
The first code works because true is a compile time constant, so the compiler is resolving it at compile time, ending up with one of the two expressions (basically removing the ternary operator). While in the second case it's a variable, thus the expression tree contains the original expression and fails at runtime due to aforementioned EF requirement.
A while ago I was trying to solve this and similar problems (to be honest, mainly for dynamic where filters) by implementing a custom method which is trying to resolve the bool variables, thus doing at runtime something similar to what does the compiler in the first case. Of course the code is experimental and not tested, but seem to handle properly such scenarios, so you can give it a try. The usage is quite simple:
var eventData = dbContext.Event.Select(t => new
{
Address = includeAddress ? new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
} : new AnonymousEventGetAddress(),
}).ReduceConstPredicates();
And here is the helper method used:
public static partial class QueryableExtensions
{
public static IQueryable<T> ReduceConstPredicates<T>(this IQueryable<T> source)
{
var visitor = new ConstPredicateReducer();
var expression = visitor.Visit(source.Expression);
if (expression != source.Expression)
return source.Provider.CreateQuery<T>(expression);
return source;
}
class ConstPredicateReducer : ExpressionVisitor
{
int evaluateConst;
private ConstantExpression TryEvaluateConst(Expression node)
{
evaluateConst++;
try { return Visit(node) as ConstantExpression; }
finally { evaluateConst--; }
}
protected override Expression VisitConditional(ConditionalExpression node)
{
var testConst = TryEvaluateConst(node.Test);
if (testConst != null)
return Visit((bool)testConst.Value ? node.IfTrue : node.IfFalse);
return base.VisitConditional(node);
}
protected override Expression VisitBinary(BinaryExpression node)
{
if (node.Type == typeof(bool))
{
var leftConst = TryEvaluateConst(node.Left);
var rightConst = TryEvaluateConst(node.Right);
if (leftConst != null || rightConst != null)
{
if (node.NodeType == ExpressionType.AndAlso)
{
if (leftConst != null) return (bool)leftConst.Value ? (rightConst ?? Visit(node.Right)) : Expression.Constant(false);
return (bool)rightConst.Value ? Visit(node.Left) : Expression.Constant(false);
}
else if (node.NodeType == ExpressionType.OrElse)
{
if (leftConst != null) return !(bool)leftConst.Value ? (rightConst ?? Visit(node.Right)) : Expression.Constant(true);
return !(bool)rightConst.Value ? Visit(node.Left) : Expression.Constant(true);
}
else if (leftConst != null && rightConst != null)
{
var result = Expression.Lambda<Func<bool>>(Expression.MakeBinary(node.NodeType, leftConst, rightConst)).Compile().Invoke();
return Expression.Constant(result);
}
}
}
return base.VisitBinary(node);
}
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (evaluateConst > 0)
{
var objectConst = node.Object != null ? TryEvaluateConst(node.Object) : null;
if (node.Object == null || objectConst != null)
{
var arguments = new object[node.Arguments.Count];
bool canEvaluate = true;
for (int i = 0; i < arguments.Length; i++)
{
var argumentConst = TryEvaluateConst(node.Arguments[i]);
if (canEvaluate = (argumentConst != null))
arguments[i] = argumentConst.Value;
else
break;
}
if (canEvaluate)
{
var result = node.Method.Invoke(objectConst != null ? objectConst.Value : null, arguments);
return Expression.Constant(result, node.Type);
}
}
}
return base.VisitMethodCall(node);
}
protected override Expression VisitUnary(UnaryExpression node)
{
if (evaluateConst > 0 && (node.NodeType == ExpressionType.Convert || node.NodeType == ExpressionType.ConvertChecked))
{
var operandConst = TryEvaluateConst(node.Operand);
if (operandConst != null)
{
var result = Expression.Lambda(node.Update(operandConst)).Compile().DynamicInvoke();
return Expression.Constant(result, node.Type);
}
}
return base.VisitUnary(node);
}
protected override Expression VisitMember(MemberExpression node)
{
object value;
if (evaluateConst > 0 && TryGetValue(node, out value))
return Expression.Constant(value, node.Type);
return base.VisitMember(node);
}
static bool TryGetValue(MemberExpression me, out object value)
{
object source = null;
if (me.Expression != null)
{
if (me.Expression.NodeType == ExpressionType.Constant)
source = ((ConstantExpression)me.Expression).Value;
else if (me.Expression.NodeType != ExpressionType.MemberAccess
|| !TryGetValue((MemberExpression)me.Expression, out source))
{
value = null;
return false;
}
}
if (me.Member is PropertyInfo)
value = ((PropertyInfo)me.Member).GetValue(source);
else
value = ((FieldInfo)me.Member).GetValue(source);
return true;
}
}
}

For future readers, this SO duplicate (added one year later) was key to solving my problems:
The type appear in two structurally incompatible initializations within a single LINQ to Entities query
When you look at it, the error message is abundantly clear. Don't mess up the initialization order if you are instantiating an object more than once in the same Linq expression. For me, that was exactly what I was doing. Synchronizing the property initializations between the two instantiation calls had the compiler blowing sunshine again.
In this case:
new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
}
is different from
new AnonymousEventGetAddress()
In OP query version 1 it is safe to say that the aberrant initialization in the else branch could never happen due to the the true conditional, why it was probably discarded, for version two that must not have happened, and we're left with two initialization orders, properties 1 and 2 versus no properties at all.
This should do it:
includeAddress
? new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
}
: new AnonymousEventGetAddress
{
AddressLine1 = null,
CityName = null
}

In some circumstances there simple workaround might be possible: make the type appear as different types. E.g. make 2 subclasses from the original class. This workaround is of course quite dirty but the Linq requirement is artificial by itself. In my case this helped.

You can put the conditional statement in each property initializer.
var eventData = dbContext.Event.Select(t => new
{
Address = new AnonymousEventGetAddress
{
AddressLine1 = includeAddress ? t.Address.AddressLine1 : null,
CityName = includeAddress ? t.Address.AddressCityName : null
}
});

In my opinion, I always try to steer away from putting anything more complicated then selecting data into IQueryable because they're Expression, which means they're never executed - they're compiled.
So, I would tackle this problem like so (this has a nice air of simplicity around it):
Create a DTO for the return data:
public class EventDto
{
// some properties here that you need
public Address Address {get;set;}
}
Then I would split your logic around the includeAddress
public IEnumerable<EventDto> IncludeAddress(DbContext dbContext)
{
return dbContext.Event.Select(t => new
{
// select out your other properties here
Address = new
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
},
}).ToList().Select(x => new EventDto { Address = Address });
// put what ever mapping logic you have up there, whether you use AutoMapper or hand map it doesn't matter.
}
The method NoAddress or whatever you want to call it will look similar but with no Address, and you map it back.
You can then choose which one simply:
var eventDtos = new List<EventDto>();
if (includeAddress)
eventDtos.AddRange(this.IncludeAddress(dbContext));
else
eventDtos.AddRange(this.NoAddress(dbContext));
eventDtos.ForEach(e => { if (e.Address == null) e.Address = new Address(); });
If you Select does have a lot of logic in it, then I would consider moving it out into a sproc where it will be easier to read the SQL.
Obviously this is just a guide, gives you an idea on how to tackle the problem.

I had the same problem, and found the solution to be, to add .ToList() , before the select-function :
var eventData = dbContext.Event.ToList().Select(t => new
{
Address = includeAddress ? new AnonymousEventGetAddress
{
AddressLine1 = t.Address.AddressLine1,
CityName = t.Address.AddressCityName
} : new AnonymousEventGetAddress(),
});

Linq query to search

I have a collection of Employee class and employee class has few properties like departement,manager name,payscale,designation etc.Now in my webapi, i have a search method in which i am searching a string across all fields of webapi
like if I search Peter it'll search in all fields(departement,manager_name,payscale,designation) of all employees.For this i am using below:-
public IEnumerable<Employee> Search(string searchstr)
{
if (repository != null)
{
var query =
from employees in repository.GetEmployees()
where
(employees.departement != null && employees.departement.Contains(searchstr)) ||
(employees.payscale != null && employees.payscale.Contains(searchstr))
(movie.designation != null && movie.designation.Contains(searchstr)) )
select employees;
return query.AsEnumerable().OrderBy(c => c.employeeid);
}
else
{
throw new HttpResponseException(new HttpResponseMessage(HttpStatusCode.NotFound));
}
}
Though i am getting desired result,i have not to use that query.Is there any other way to rewrite same query?

As stated by Noctis, using reflection results in a heavy task for the .NET runtime.
Here is some example code that loops trough all properties of a class and searchs a string concidence. It uses reflection ;)
Any questions, leave a comment asking!
The code in the entry point of the APP:
[STAThread]
private static void Main(string[] args)
{
var person1 = new Person {Name = "The first name", Address = "The first address"};
var person2 = new Person {Name = "The second name", Address = "The second address"};
var results = SearchStringTroughAllProperties("second", new List<Person> {person1, person2});
}
The Person class:
class Person
{
public string Name { get; set; }
public string Address { get; set; }
}
And the SearchStringTroughAllProperties method:
private static IEnumerable<Person> SearchStringTroughAllProperties(string stringToSearch,
IEnumerable<Person> persons)
{
var properties =
typeof (Person).GetProperties()
.Where(x => x.CanRead && x.PropertyType == typeof (string))
.Select(x => x.GetMethod)
.Where(x => !x.IsStatic)
.ToList();
return persons.Where(person =>
properties.Select(property => (string) property.Invoke(person, null) ?? string.Empty)
.Any(propertyValueInInstance => propertyValueInInstance.Contains(stringToSearch)));
}
Notice that:
It searchs in properties, not in fields
It only searchs on properties that can be read (have a get defined)
It only searchs on properties of type string
It only searchs on properties that aren't static members of the class
EDIT:
For searching the string coincidence in a string or string[] property, change SearchStringTroughAllProperties method to this (it gets longer!):
static IEnumerable<Person> SearchStringTroughAllProperties(string stringToSearch, IEnumerable<Person> persons)
{
var properties =
typeof (Person).GetProperties()
.Where(x => x.CanRead && (x.PropertyType == typeof (string) || x.PropertyType == typeof(string[])))
.Select(x => x.GetMethod)
.Where(x => !x.IsStatic)
.ToList();
foreach (var person in persons)
{
foreach (var property in properties)
{
bool yieldReturned = false;
switch (property.ReturnType.ToString())
{
case "System.String":
var propertyValueStr = (string) property.Invoke(person, null) ?? string.Empty;
if (propertyValueStr.Contains(stringToSearch))
{
yield return person;
yieldReturned = true;
}
break;
case "System.String[]":
var propertyValueStrArr = (string[]) property.Invoke(person, null);
if (propertyValueStrArr != null && propertyValueStrArr.Any(x => x.Contains(stringToSearch)))
{
yield return person;
yieldReturned = true;
}
break;
}
if (yieldReturned)
{
break;
}
}
}
}

Even though work, it feels a bit dirty. I would consider maybe using reflection to get the properties of the class, and then dynamically search them.
The benefit would be: if you add a new property tomorrow, there's nothing else you need to change.
The disadvantage would be: probably not as performant since reflection is much slower than simply searching for things you know exist.
Having said that, i'm sure there are some other nifty advanced linq tricks that maybe others can point out.
Accessing Attributes by Using Reflection (C# and Visual Basic)
I thought I have some handy code but I don't. I wouldn't like to write it of the top of my head, because it probably won't compile (you need to get the syntax right).
Have a look at the above link :)

Lambda How to reordering elements in list ascending order and place null value at the back?

Says I have ListA={null,3,2,null}.
ListA.OrderBy(x=>x.ID) //would return me null,null,2,3
If my objective is to get 2,3,null,null, currently I can only think of extracting out the null item, and manually pump into the back.
Is there a clean approach where it will return me 2,3,null,null?

You can use OrderByDescending + ThenBy(assuming that it's aList<int?>):
var orderedList = ListA
.OrderByDescending(x => x.HasValue)
.ThenBy(x => x);
x.HasValue returns true or false where true is higher than false. That's why i'm using OrderByDescending.
If you want to sort the original list i would use List.Sort with a custom Compaison<T> that treats null as highest value:
ListA.Sort((a1, a2) => (a1 ?? int.MaxValue).CompareTo(a2 ?? int.MaxValue));
This is more efficient since it doesn't need to create a new list.

As an alternative to Tim's answer you could write your own IComparer<T> which does the custom sorting algorithm for you.
var array = list.OrderBy(x => x, new NullableIntComparer())
.ToArray();
class NullableIntComparer : IComparer<int?>
{
public int Compare(int? x, int? y)
{
if (x.HasValue && y.HasValue)
{
return x.Value.CompareTo(y.Value);
}
if (x.HasValue)
{
return -1;
}
if (y.HasValue)
{
return 1;
}
return 0;
}
}

Tried the following:
class Program
{
class A
{
public A(){}
public int? ID { get; set; }
}
static void Main(string[] args)
{
var listA = new List<A>
{
new A(){ID = null},
new A(){ID = 2},
new A(){ID = null},
new A(){ID = 3},
};
var result = listA.OrderByDescending(x => x.ID != null).ThenBy(x => x.ID);
foreach (var a in result)
{
Console.WriteLine(a.ID);
}
}
}

LINQ query to perform a projection, skipping or wrapping exceptions where source throws on IEnumerable.GetNext()

I'd like a general solution but as an example, assume i have an IEnumerable<string>, where some can be parsed as integers, and some cannot.
var strings = new string[] { "1", "2", "notint", "3" };
Obviously if i did Select(s => int.Parse(s, temp)) it'd throw an exception when enumerated.
In this case i could do .All(s => int.TryParse(s, out temp)) first, however i want a general solution where i don't have to enumerate the IEnumerable twice.
Ideally i'd like to be able to do the following, which calls my magic exception skipping method:
// e.g. parsing strings
var strings = new string[] { "1", "2", "notint", "3" };
var numbers = strings.Select(s => int.Parse(s)).SkipExceptions();
// e.g. encountering null object
var objects = new object[] { new object(), new object(), null, new object() }
var objecttostrings = objects.Select(o => o.ToString()).SkipExceptions();
// e.g. calling a method that could throw
var myClassInstances = new MyClass[] { new MyClass(), new MyClass(CauseMethodToThrow:true) };
var myClassResultOfMethod = myClassInstances.Select(mci => mci.MethodThatCouldThrow()).SkipExceptions();
How can i write the SkipExceptions() extension method?
Some great answers for a SelectSkipExceptions() method, however i wonder if a SkipExceptions() method could be created, along the same lines as AsParallel().

How about this (you might want to give this special Select Extension a better name)
public static IEnumerable<TOutput> SelectIgnoringExceptions<TInput, TOutput>(
this IEnumerable<TInput> values, Func<TInput, TOutput> selector)
{
foreach (var item in values)
{
TOutput output = default(TOutput);
try
{
output = selector(item);
}
catch
{
continue;
}
yield return output;
}
}
Edit5
Added a using statement, thanks for the suggestion in comments
public static IEnumerable<T> SkipExceptions<T>(
this IEnumerable<T> values)
{
using(var enumerator = values.GetEnumerator())
{
bool next = true;
while (next)
{
try
{
next = enumerator.MoveNext();
}
catch
{
continue;
}
if(next) yield return enumerator.Current;
}
}
}
However this relies on the incoming IEnumerable not already being created (and therefore already having thrown Exceptions) as a list by the preceding Function. E.g. this would probably not work if you call it like this: Select(..).ToList().SkipExceptions()

Create a TryParseInt method that returns a Nullable<int>:
int? TryParseInt(string s)
{
int i;
if (int.TryParse(s, out i))
return i;
return null;
}
And use it in your query like that:
var numbers = strings.Select(s => TryParseInt(s))
.Where(i => i.HasValue)
.Select(i => i.Value);
See also this article by Bill Wagner, which presents a very similar case.
Now, i don't think you can write something like a generic SkipExceptions method, because you would catch the exception too late, and it would end the Select loop... But you could probably write a SelectSkipException method:
public static IEnumerable<TResult> SelectSkipExceptions<TSource, TResult>(
this IEnumerable<TSource> source,
Func<TSource, TResult> selector)
{
if (source == null)
throw new ArgumentNullException("source");
if (selector == null)
throw new ArgumentNullException("selector");
return source.SelectSkipExceptionsIterator(selector);
}
private static IEnumerable<TResult> SelectSkipExceptionsIterator<TSource, TResult>(
this IEnumerable<TSource> source,
Func<TSource, TResult> selector)
{
foreach(var item in source)
{
TResult value = default(TResult);
try
{
value = selector(item);
}
catch
{
continue;
}
yield return value;
}
}

Even the accepted answer may not be "general" enough. What if some day you find that you need to know what exceptions occurred?
The following extension
static class EnumeratorHelper {
//Don't forget that GetEnumerator() call can throw exceptions as well.
//Since it is not easy to wrap this within a using + try catch block with yield,
//I have to create a helper function for the using block.
private static IEnumerable<T> RunEnumerator<T>(Func<IEnumerator<T>> generator,
Func<Exception, bool> onException)
{
using (var enumerator = generator())
{
if (enumerator == null)
yield break;
for (; ; )
{
//You don't know how to create a value of T,
//and you don't know weather it can be null,
//but you can always have a T[] with null value.
T[] value = null;
try
{
if (enumerator.MoveNext())
value = new T[] { enumerator.Current };
}
catch (Exception e)
{
if (onException(e))
continue;
}
if (value != null)
yield return value[0];
else
yield break;
}
}
}
public static IEnumerable<T> WithExceptionHandler<T>(this IEnumerable<T> orig,
Func<Exception, bool> onException)
{
return RunEnumerator(() =>
{
try
{
return orig.GetEnumerator();
}
catch (Exception e)
{
onException(e);
return null;
}
}, onException);
}
}
will help. Now you can add SkipExceptions:
public static IEnumerable<T> SkipExceptions<T>(this IEnumerable<T> orig){
return orig.WithExceptionHandler(orig, e => true);
}
By using different onException callback, you can do different things
Break the iteration but ignore the exception: e => false
Try to continue iteration: e => true
Log the exception, etc

Here's a small complete program to demonstrate an answer inspired by the maybe monad. You might want to change the name of the 'Maybe' class, as it is inspired by rather than actually being a 'Maybe' as defined in other languages.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace TestMaybe
{
class Program
{
static void Main(string[] args)
{
var strings = new string[] { "1", "2", "notint", "3" };
var ints = strings.Select(s => new Maybe<string, int>(s, str => int.Parse(str))).Where(m => !m.nothing).Select(m => m.value);
foreach (var i in ints)
{
Console.WriteLine(i);
}
Console.ReadLine();
}
}
public class Maybe<T1, T2>
{
public readonly bool nothing;
public readonly T2 value;
public Maybe(T1 input, Func<T1, T2> map)
{
try
{
value = map(input);
}
catch (Exception)
{
nothing = true;
}
}
}
}
Edit: depending on the needs of your code, you might also want nothing set to true if the result of map(input) is null.

This is the same answer as Thomas's, but with a lambda & LINQ expression. +1 for Thomas.
Func<string, int?> tryParse = s =>
{
int? r = null;
int i;
if (int.TryParse(s, out i))
{
r = i;
}
return r;
};
var ints =
from s in strings
let i = tryParse(s)
where i != null
select i.Value;

You could just chain the Where and Select method together.
var numbers = strings.Where(s =>
{
int i;
return int.TryParse(s, out i);
}).Select(int.Parse);
The use of the Where method effectively removes the need for you to write your own SkipExceptions method, because this is basically what you are doing.