We Keep Coding

Lambda .Where limitation to string

I have this method:
public List<object> GetThings(List<Guid> listOfGuids)
{
var query = serviceContext.Xrm.crmEntity;
bool anyTypeOfSearch = false; // use this to know if we have actually applied any search criteria.
if(listOfGuids != null && listOfGuids.Count > 0)
{
query = query.Where(x => listOfGuids.Contains(x.lgc_muncipalityid.Id));
anyTypeOfSearch = true;
}
var result = new List<object>();
if(anyTypeOfSearch) // instead of a variable here, can i check if there are any whereconditions applied to the query?
result = query
.Select(x => new SupplierSearchResultModel()
{
Id = x.Id,
Name = x.lgc_name,
})
.ToList();
LogMessage("GetThings.Query", <insert code to get query.Where condition tostring()>);
return result;
}
In the real code there are several different if structures with .Where conditions in them and sometimes a call can reach this code without any parameters. In this case I don't want to run the query as the result set would be huge. So I only want to run the query if at least once the .Where() condition has been applied.
Now my question is, can I check a lambda query variable for if it has any .Where() conditions applied without using an external bool like I am?
An alternate interesting usage point would be if there is some way to get some sort of query.Where().ToString() method that would show what conditions will be applied which could be logged in case of errors...

Quick & dirty, if you don't care about having a pretty result:
LogMessage(query.Expression.ToString());
But it will not show you the content of your array parameter, though.
edit Better solutions:
1) What you are looking for is an expression visitor. A template for what you want to do here, which should then be used like:
LogMessage(query.ToPrettyString());
2) Think about an expression query.Where(x=>x.member == GetSomething()) do you want it to be printed like that ? Or do you want GetSomething() result to appear as a string result ? If the second solution, then that's something you can do with this

You can create your own implementation of the ExpressionVisitor to traverse the nodes of the expression. You can do something like this:
public class WhereVisitor : ExpressionVisitor
{
private static bool _filter;
private static WhereVisitor _visitor = new WhereVisitor();
private WhereVisitor() { }
public new static bool Visit(Expression expression)
{
_filter = false;
//Cast to ExpressionVisitor to use the default Visit and not our new one
((ExpressionVisitor)_visitor).Visit(expression);
return _filter;
}
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.Name == "Where")
_filter = true;
return base.VisitMethodCall(node);
}
}
And use it like this:
bool containsWhere = WhereVisitor.Visit(query.Expression);
If you want you can of course expand the visitor to save the expressions that contain a Where clause, but this one will just tell you if there are is Where or not.

LINQ with Querying "Memory"

Does LINQ have a way to "memorize" its previous query results while querying?
Consider the following case:
public class Foo {
public int Id { get; set; }
public ICollection<Bar> Bars { get; set; }
}
public class Bar {
public int Id { get; set; }
}
Now, if two or more Foo have same collection of Bar (no matter what the order is), they are considered as similar Foo.
Example:
foo1.Bars = new List<Bar>() { bar1, bar2 };
foo2.Bars = new List<Bar>() { bar2, bar1 };
foo3.Bars = new List<Bar>() { bar3, bar1, bar2 };
In the above case, foo1 is similar to foo2 but both foo1 and foo2 are not similar tofoo3
Given that we have a query result consisting IEnumerable or IOrderedEnumerable of Foo. From the query, we are to find the first N foo which are not similar.
This task seems to require a memory of the collection of bars which have been chosen before.
With partial LINQ we could do it like this:
private bool areBarsSimilar(ICollection<Bar> bars1, ICollection<Bar> bars2) {
return bars1.Count == bars2.Count && //have the same amount of bars
!bars1.Select(x => x.Id)
.Except(bars2.Select(y => y.Id))
.Any(); //and when excepted does not return any element mean similar bar
}
public void somewhereWithQueryResult(){
.
.
List<Foo> topNFoos = new List<Foo>(); //this serves as a memory for the previous query
int N = 50; //can be any number
foreach (var q in query) { //query is IOrderedEnumerable or IEnumerable
if (topNFoos.Count == 0 || !topNFoos.Any(foo => areBarsSimilar(foo.Bars, q.Bars)))
topNFoos.Add(q);
if (topNFoos.Count >= N) //We have had enough Foo
break;
}
}
The topNFoos List will serve as a memory of the previous query and we can skip the Foo q in the foreach loop which already have identical Bars with Any of the Foo in the topNFoos.
My question is, is there any way to do that in LINQ (fully LINQ)?
var topNFoos = from q in query
//put something
select q;
If the "memory" required is from a particular query item q or a variable outside of the query, then we could use let variable to cache it:
int index = 0;
var topNFoos = from q in query
let qc = index++ + q.Id //depends on q or variable outside like index, then it is OK
select q;
But if it must come from the previous querying of the query itself then things start to get more troublesome.
Is there any way to do that?
Edit:
(I currently am creating a test case (github link) for the answers. Still figuring out how can I test all the answers fairly)
(Most of the answers below are aimed to solve my particular question and are in themselves good (Rob's, spender's, and David B's answers which use IEqualityComparer are particularly awesome). Nevertheless, if there is anyone who can give answer to my more general question "does LINQ have a way to "memorize" its previous query results while querying", I would also be glad)
(Apart from the significant difference in performance for the particular case I presented above when using fully/partial LINQ, one answer aiming to answer my general question about LINQ memory is Ivan Stoev's. Another one with good combination is Rob's. As to make myself clearer, I look for general and efficient solution, if there is any, using LINQ)

I'm not going to answer your question directly, but rather, propose a method that will be fairly optimally efficient for filtering the first N non-similar items.
First, consider writing an IEqualityComparer<Foo> that uses the Bars collection to measure equality. Here, I'm assuming that the lists might contain duplicate entries, so have quite a strict definition of similarity:
public class FooSimilarityComparer:IEqualityComparer<Foo>
{
public bool Equals(Foo a, Foo b)
{
//called infrequently
return a.Bars.OrderBy(bar => bar.Id).SequenceEqual(b.Bars.OrderBy(bar => bar.Id));
}
public int GetHashCode(Foo foo)
{
//called frequently
unchecked
{
return foo.Bars.Sum(b => b.GetHashCode());
}
}
}
You can really efficiently get the top N non-similar items by using a HashSet with the IEqualityComparer above:
IEnumerable<Foo> someFoos; //= some list of Foo
var hs = new HashSet<Foo>(new FooSimilarityComparer());
foreach(var f in someFoos)
{
hs.Add(f); //hashsets don't add duplicates, as measured by the FooSimilarityComparer
if(hs.Count >= 50)
{
break;
}
}
#Rob s approach above is broadly similar, and shows how you can use the comparer directly in LINQ, but pay attention to the comments I made to his answer.

So, it's ... possible. But this is far from performant code.
var res = query.Select(q => new {
original = q,
matches = query.Where(innerQ => areBarsSimilar(q.Bars, innerQ.Bars))
}).Select(g => new { original = g, joinKey = string.Join(",", g.matches.Select(m => m.Id)) })
.GroupBy (g => g.joinKey)
.Select(g => g.First().original.original)
.Take(N);
This assumes that the Ids are unique for each Foo (you could also use their GetHashCode(), I suppose).
A much better solution is to either keep what you've done, or implement a custom comparer, as follows:
Note: As pointed out in the comments by #spender, the below Equals and GetHashCode will not work for collections with duplicates. Refer to their answer for a better implementation - however, the usage code would remain the same
class MyComparer : IEqualityComparer<Foo>
{
public bool Equals(Foo left, Foo right)
{
return left.Bars.Count() == right.Bars.Count() && //have the same amount of bars
left.Bars.Select(x => x.Id)
.Except(right.Bars.Select(y => y.Id))
.ToList().Count == 0; //and when excepted returns 0, mean similar bar
}
public int GetHashCode(Foo foo)
{
unchecked {
int hc = 0;
if (foo.Bars != null)
foreach (var p in foo.Bars)
hc ^= p.GetHashCode();
return hc;
}
}
}
And then your query becomes simply:
var res = query
.GroupBy (q => q, new MyComparer())
.Select(g => g.First())
.Take(N);

IEnumerable<Foo> dissimilarFoos =
from foo in query
let key = string.Join('|',
from bar in foo.Bars
order by bar.Id
select bar.Id.ToString())
group foo by key into g
select g.First();
IEnumerable<Foo> firstDissimilarFoos =
dissimilarFoos.Take(50);
Sometimes, you may not like the behavior of groupby in the above queries. At the time the query is enumerated, groupby will enumerate the entire source. If you only want partial enumeration, then you should switch to Distinct and a Comparer:
class FooComparer : IEqualityComparer<Foo>
{
private string keyGen(Foo foo)
{
return string.Join('|',
from bar in foo.Bars
order by bar.Id
select bar.Id.ToString());
}
public bool Equals(Foo left, Foo right)
{
if (left == null || right == null) return false;
return keyGen(left) == keyGen(right);
}
public bool GetHashCode(Foo foo)
{
return keyGen(foo).GetHashCode();
}
}
then write:
IEnumerable<Foo> dissimilarFoos = query.Distinct(new FooComparer());
IEnumerable<Foo> firstDissimilarFoos = dissimilarFoos.Take(50);

Idea. You might be able to hack something by devising your own fluent interface of mutators over a cache that you'd capture in "let x = ..." clauses, along the lines of,
from q in query
let qc = ... // your cache mechanism here
select ...
but I suspect you'll have to be careful to limit the updates to your cache to those "let ..." only, as I doubt the implementation of the standard Linq operators and extensions methods will be happy if you allow such side effects to happen in their back through predicates applied in the "where", or "join", "group by", etc, clauses.
'HTH,

I guess by "full LINQ" you mean standard LINQ operators/Enumerable extension methods.
I don't think this can be done with LINQ query syntax. From standard methods the only one that supports mutable processing state is Enumerable.Aggregate, but it gives you nothing more than a LINQ flavor over the plain foreach:
var result = query.Aggregate(new List<Foo>(), (list, next) =>
{
if (list.Count < 50 && !list.Any(item => areBarsSimilar(item.Bars, next.Bars)))
list.Add(next);
return list;
});
Since looks like we are allowed to use helper methods (like areBarsSimilar), the best we can do is to make it at least look more LINQ-ish by defining and using a custom extension method
var result = query.Aggregate(new List<Foo>(), (list, next) => list.Count < 50 &&
!list.Any(item => areBarsSimilar(item.Bars, next.Bars)) ? list.Concat(next) : list);
where the custom method is
public static class Utils
{
public static List<T> Concat<T>(this List<T> list, T item) { list.Add(item); return list; }
}
But note that compared to vanilla foreach, Aggregate has an additional drawback of not being able to exit earlier, thus will consume the whole input sequence (which besides the performance also means it doesn't work with infinite sequences).
Conclusion: While this should answer your original question, i.e. it's technically possible to do what you are asking for, LINQ (like the standard SQL) is not well suited for such type of processing.

How do I return a different value from a stub based on a lambda argument

I have the following sample test code
public Stage Test(Stage Stage)
{
var StartStage = StageRepository.Single(x => x.Order == 1);
var EndStage = StageRepository.Single(x => x.Order == 5);
var ErrorStage = StageRepository.Single(x => x.Name == "Error");
if (Stage == StartStage)
{
return EndStage;
}
else
{
return ErrorStage;
}
}
And I am trying to test it using the following unit test
[TestMethod]
public void XXXTest()
{
//// Arrange
var AutoMocker = new RhinoAutoMocker<StageService>(MockMode.AAA);
MockRepository mockRepository = new MockRepository();
var MockStageRepository = AutoMocker.Get<IRepository<Stage>>();
Stage StartStage = mockRepository.Stub<Stage>();
StartStage.Order = 1;
Stage EndStage = mockRepository.Stub<Stage>();
EndStage.Order = 5;
Stage ErrorStage = mockRepository.Stub<Stage>();
ErrorStage.Name = "Error";
System.Linq.Expressions.Expression<Func<Entities.Stage, bool>> StartParam = x => x.Order == 1;
MockStageRepository
.Stub(x => x.Single(Arg<System.Linq.Expressions.Expression<Func<Entities.Stage, bool>>>.Is.Equal(StartParam)))
.Return(StartStage);
System.Linq.Expressions.Expression<Func<Entities.Stage, bool>> EndParam = x => x.Order == 1;
MockStageRepository
.Stub(x => x.Single(Arg<System.Linq.Expressions.Expression<Func<Entities.Stage, bool>>>.Is.Equal(EndParam)))
.Return(EndStage);
System.Linq.Expressions.Expression<Func<Entities.Stage, bool>> ErrorParam = x => x.Order == 1;
MockStageRepository
.Stub(x => x.Single(Arg<System.Linq.Expressions.Expression<Func<Entities.Stage, bool>>>.Is.Equal(ErrorParam)))
.Return(ErrorStage);
StageService StageService = AutoMocker.ClassUnderTest;
//Act
var ReturnStage = StageService.Test(StartStage);
//Assert
Assert.AreEqual(ReturnStage, EndStage);
}
However this is not working as it is not returning anything when I call StageRepository.Single(). If I change the stub code to ignore the argument then it does return something but it will be the same object returned for each call to Single() which I don't want.
Is it possible to configure RhinoMocks in such a way as to return different objects from a stub depending on the lambda that is passed into it?

I think the root of your problem is that equality on the Expression<Func<T,U>> type is performed by reference rather than value. That means your telling Rhino Mocks to look for instances of expressions created in the test rather than the ones created in the code your testing.
Two possible approaches come to mind:
One would be to provide a way to pass the lambda expressions in to the Stage class from the test, so that when the argument checks happen they are working against the same instances.
Maybe something like:
internal void SetStartStage(Expression<Func<Entities.Stage,bool>> predicate)
{
...
}
The inverse of this would also work, i.e. provide the Expression objects as fields/properties that can be accessed by you're test, and then use those when setting up your mock:
internal Expression<Func<Entities.State,bool>> StartStagePredicate
{
get{ return x => x.Order == 1; }
}
Another option would be to use the Matches method on Args to see if the argument checks the Stage object state correctly. This would require creating some Stage objects in your test that would match the criteria:
var startStageTester = new Stage { Order = 1 };
MockStageRepository
.Stub(x => x.Single(Arg<System.Linq.Expressions.Expression<Func<Entities.Stage, bool>>>.Matches(y => y.Compile()(startStageTester)))
.Return(StartStage);
The call to Compile() is a little jarring, but since you're dealing with an Expression and not a straight-up lambda, you've got to compile it in order to evaluate it.
Now, if a Stage is something that is hard to create, then you may need to create a Mock/Stub of one (or use you're StartStage) and have it return 1 from the Order property (in the case of the StartStage anyway).
There are probably some others I'm not thinking of at the moment as well.

IF the linq condition is not important, I would propose to match it as any, something like this
repository.Stub(x => x.Find(Arg<System.Linq.Expressions.Expression<Func<Entity, bool>>>.Is.Anything)).Return(entitiesList.AsQueryable());
Hope this help.

Thanks for that. That worked a treat. The one thing that I was missing was the Compile method on an ExpressionTree. There is a small bug in your sample code (you use x twice). For completeness here is the working code.
MockStageRepository2
.Stub(x => x.Single(Arg<System.Linq.Expressions.Expression<Func<Entities.Stage, bool>>>.Matches(y => y.Compile()(StartStage2))))
.Return(StartStage2);

Compose LINQ-to-SQL predicates into a single predicate

(An earlier question, Recursively (?) compose LINQ predicates into a single predicate, is similar to this but I actually asked the wrong question... the solution there satisfied the question as posed, but isn't actually what I need. They are different, though. Honest.)
Given the following search text:
"keyword1 keyword2 ... keywordN"
I want to end up with the following SQL:
SELECT [columns] FROM Customer
WHERE (
Customer.Forenames LIKE '%keyword1%'
OR
Customer.Forenames LIKE '%keyword2%'
OR
...
OR
Customer.Forenames LIKE '%keywordN%'
) AND (
Customer.Surname LIKE '%keyword1%'
OR
Customer.Surname LIKE '%keyword2%'
OR
....
OR
Customer.Surname LIKE '%keywordN%'
)
Effectively, we're splitting the search text on spaces, trimming each token, constructing a multi-part OR clause based on each , and then AND'ing the clauses together.
I'm doing this in Linq-to-SQL, and I have no idea how to dynamically compose a predicate based on an arbitrarily-long list of subpredicates. For a known number of clauses, it's easy to compose the predicates manually:
dataContext.Customers.Where(
(
Customer.Forenames.Contains("keyword1")
||
Customer.Forenames.Contains("keyword2")
) && (
Customer.Surname.Contains("keyword1")
||
Customer.Surname.Contains("keyword2")
)
);
In short, I need a technique that, given two predicates, will return a single predicate composing the two source predicates with a supplied operator, but restricted to the operators explicitly supported by Linq-to-SQL. Any ideas?

You can use the PredicateBuilder class
IQueryable<Customer> SearchCustomers (params string[] keywords)
{
var predicate = PredicateBuilder.False<Customer>();
foreach (string keyword in keywords)
{
// Note that you *must* declare a variable inside the loop
// otherwise all your lambdas end up referencing whatever
// the value of "keyword" is when they're finally executed.
string temp = keyword;
predicate = predicate.Or (p => p.Forenames.Contains (temp));
}
return dataContext.Customers.Where (predicate);
}
(that's actually the example from the PredicateBuilder page, I just adapted it to your case...)
EDIT:
Actually I misread your question, and my example above only covers a part of the solution... The following method should do what you want :
IQueryable<Customer> SearchCustomers (string[] forenameKeyWords, string[] surnameKeywords)
{
var predicate = PredicateBuilder.True<Customer>();
var forenamePredicate = PredicateBuilder.False<Customer>();
foreach (string keyword in forenameKeyWords)
{
string temp = keyword;
forenamePredicate = forenamePredicate.Or (p => p.Forenames.Contains (temp));
}
predicate = PredicateBuilder.And(forenamePredicate);
var surnamePredicate = PredicateBuilder.False<Customer>();
foreach (string keyword in surnameKeyWords)
{
string temp = keyword;
surnamePredicate = surnamePredicate.Or (p => p.Surnames.Contains (temp));
}
predicate = PredicateBuilder.And(surnamePredicate);
return dataContext.Customers.Where(predicate);
}
You can use it like that:
var query = SearchCustomers(
new[] { "keyword1", "keyword2" },
new[] { "keyword3", "keyword4" });
foreach (var Customer in query)
{
...
}

Normally you would chain invocations of .Where(...). E.g.:
var a = dataContext.Customers;
if (kwd1 != null)
a = a.Where(t => t.Customer.Forenames.Contains(kwd1));
if (kwd2 != null)
a = a.Where(t => t.Customer.Forenames.Contains(kwd2));
// ...
return a;
LINQ-to-SQL would weld it all back together into a single WHERE clause.
This doesn't work with OR, however. You could use unions and intersections, but I'm not sure whether LINQ-to-SQL (or SQL Server) is clever enough to fold it back to a single WHERE clause. OTOH, it won't matter if performance doesn't suffer. Anyway, it would look something like this:
<The type of dataContext.Customers> ff = null, ss = null;
foreach (k in keywords) {
if (keywords != null) {
var f = dataContext.Customers.Where(t => t.Customer.Forenames.Contains(k));
ff = ff == null ? f : ff.Union(f);
var s = dataContext.Customers.Where(t => t.Customer.Surname.Contains(k));
ss = ss == null ? s : ss.Union(s);
}
}
return ff.Intersect(ss);

How can I create a dynamic select expression in LINQ?

What I currently have looks a bit like this:
if(userLikesBananas)
{
return from fruit in basket
select new Fruit
{
AteBanana = Bowl.Any(b => b.OwnedBy == user && b.Contains(fruit) && fruit.Type == FruitType.Banana),
...
...
//lots of properties
...
}
}
else
{
return from fruit in basket
select new Fruit
{
AteBanana = Bowl.Any(b => b.Contains(fruit)),
...
...
//lots of properties
...
}
}
Admittedly the example makes absolutely no sense, but the principle is that I want to change the conditions of a properties selection based on arbitrary criteria. Right now the select statements are repeated.
Now the time has come that I need to add anoter dependent criteria. I don't want to have 4 different cases where the property conditions are slightly different.
What I want to do, is something like this:
Func<Fruit, bool> fruitFunc = f => false;
if(userLikesBananas)
{
fruitFunc = f => Bowl.Any(b => b.OwnedBy == user && b.Contains(f) && f.Type == FruitType.Banana);
}
else
{
fruitFunc = f => Bowl.Any(b => b.Contains(f));
}
return from fruit in basket
select new Fruit
{
AteBanana = fruitFunc(fruit)
...
...
//lots of properties
...
};
The trouble is that is the expression cannot be converted to sql, as it contains a dynamic invoke. I have tried wrapping the Func in an Expression, but the same problem seems to arise.
So the question is, how can I avoid the copy and paste?

...my english is not good, but I'll try to explain how to easily solve this problem :-)
Dynamic Linq is bad for type control - It's easy to use, but you can't browse the resulting object (x.Name, x.Surname, etc.)
There is (a litle noob-like) trick to solve this problem (I'm using it and its working fain):
Create enum with attributes of your object you're selecting from.
public enum MyAtrs{ID, FirstName, Surname}
create Dictionary<MyAtrs,bool> (and fill it) for conditions (set true if you want to get this property)
public Dictionary Dic = new Dictionary();
Dic.Add(MyAtrs.ID,true);
Dic.Add(MyAtrs.Firstname,false);
Dic.Add(MyAtrs.Surname,true);
Built your query:
var query = DBContext.MyDBTable.Where(predicate).Select(e=>new {
ID = Dic[MyAtrs.ID] ? e.dbID:0,
Firstname = Dic[MyAtrs.Firstname] ? e.dbFirstname:null,
Surname = Dic[MyAtrs.Surname] ? e.dbSurname:null,
});
In this case, there will be all 3 columns in the SQL Select statement, but it's just a few bytes (who matters...). SQL Server gives you back all 3 columns but (in this case) Firstname will be empty (something like {ID=123, Firstname=, Surname="Jobs"}).
It's not brilliant but it's easy way how to built "dynamic" select expression without loosing type control :)

I can suggest writing the Func in a different way (maybe something like this):
fruitFunc = f => Bowl.Any(b => ((f.Type == FruitType.Banana && b.OwnedBy == user && userLikesBananas) || !userLikesBananas) && b.Contains(f));
I haven't tested if it works but this may be a way to write the function to cover more cases (not only 2). The approach is somehow similar to boolean algebra as I remember...
Cheers...