I have a method where I am trying to return all default customer addresses with the matching gender. I would like to be able to build up the filtering query bit by bit by passing in System.Func methods to the where clause.
var emailAddresses = new List<string>();
// get all customers.
IQueryable<Customer> customersQ = base.GetAllQueryable(appContext).Where(o => o.Deleted == false);
// for each customer filter, filter the query.
var genders = new List<string>() { "C" };
Func<Customer, bool> customerGender = (o => genders.Contains(o.Addresses.FirstOrDefault(a => a.IsDefaultAddress).Gender));
customersQ = customersQ.Where(customerGender).AsQueryable();
emailAddresses = (from c in customersQ
select c.Email).Distinct().ToList();
return emailAddresses;
But this method calls the database for every address (8000) times which is very slow.
however if I replace the two lines
Func<Customer, bool> customerGender = (o => genders.Contains(o.Addresses.FirstOrDefault(a => a.IsDefaultAddress).Gender));
customersQ = customersQ.Where(customerGender).AsQueryable();
with one line
customersQ = customersQ.Where(o => genders.Contains(o.Addresses.FirstOrDefault(a => a.IsDefaultAddress).Gender)).AsQueryable();
Then the query only makes one call to the database and is very fast.
My question is why does this make a difference? How can I make the first method work with only calling the database once?
Use expression instead of Func:
Expression<Func<Customer, bool>> customerGender = (o =>
genders.Contains(o.Addresses.FirstOrDefault(a => a.IsDefaultAddress).Gender));
customersQ = customersQ.Where(customerGender).AsQueryable();
When you are using simple Func delegate, then Where extension of Enumerable is called. Thus all data goes into memory, where it is enumerated and lambda is executed for each entity. And you have many calls to database.
On the other hand, when you are using expression, then Where extension of Queryable is called, and expression is converted into SQL query. That's why you have single query in second case (if you use in-place lambda it is converted into expression).
Related
I have this idea to create a "list" of IQueryables that do different kinds of operations.
So basically:
var query1 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Name == "Ronald");
var query2 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Age == 43);
var query3 = Enumerable.Empty<Person>().AsQueryable().Select(e => e.EyeColor);
var listOfQueries = new List<IQueryable<Person>
{
query1,
query2,
query3
};
Now, I also have this DbSet full of "Persons" and I would like to "apply" all my queries against that DbSet. How would I do that? Is it even possible?
An updated example:
var personQueryFactory = new PersonQueryFactory();
var personQueryByFirstname = personQueryFactory.CreateQueryByFirstname("Ronald"); //Query by Firstname.
var personQueryByAge = personQueryFactory.CreateQueryByAge(42); //Query by age.
var personQueryByHasChildWithAgeOver = personQueryFactory.CreateQueryByChildAgeOver(25); //Query using a "join" to the child-relationship.
var personQuerySkip = personQueryFactory.Take(5); //Only get the 5 first matching the queries.
var personQuery = personQueryFactory.AggregateQueries //Aggragate all the queries into one single query.
(
personQueryByFirstname,
personQueryByAge,
personQueryByHasChildWithAgeOver,
personQuerySkip
);
var personSurnames = personsService.Query(personQuery, e => new { Surname = e.Surname }); //Get only the surname of the first 5 persons with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
var personDomainObjects = personsService.Query<DomainPerson>(personQuery); //Get the first 5 persons as a domain-object (mapping behind the "scenes") with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
var personDaos = personsService.Query(personQuery); //Get the first 5 persons as a DAO-objects/entityframework-entities with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
The reason for doing this would be to create a more "unified" way of creating and re-using predefined queries, and then to be able to execute them against the DbSet and return the result as a domain-object and not an "entity-framework-model/object"
This is possible, but you need to reframe your approach slightly.
Storing the filters
var query1 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Name == "Ronald");
var query2 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Age == 43);
var query3 = Enumerable.Empty<Person>().AsQueryable().Select(e => e.EyeColor);
Reading your intention, you don't actually want to handle IQueryable objects, but rather the parameter that you supply to the Where method.
Edit: I missed that the third one was a Select(), not a Where(). Ive adjusted the rest of the answer as if this had also been a Where(). See the comments below for my response as to why you can't mix Select() with Where() easily.
Func<Person,bool> filter1 = (e => e.Name == "Ronald");
Func<Person,bool> filter2 = (e => e.Age == 43);
Func<Person,bool> filter3 = (e => e.EyeColor == "Blue");
This stores the same information (filter criteria), but it doesn't wrap each filter in an IQueryable of its own.
A short explanation
Notice the Func<A,B> notation. In this case, A is the input type (Person), and B is the output type (bool).
This can be extended further. A Func<string,Person,bool> has two input parameters (string, Person) and one output parameter (bool). A usage example:
Func<string, Person, bool> filter = (inputString, inputPerson) => inputString == "TEST" && inputPerson.Age > 35;
There is always one output parameter (the last type). Every other mentioned type is an input parameter.
Putting the filters in a list
Intuitively, since Func<Person,bool> represents a single filter; you can represent a list of filters by using a List<Func<Person,bool>>.
Nesting generic types get a bit hard to read, but it does work just like any other List<T>.
List<Func<Person,bool>> listOfFilters = new List<Func<Person,bool>>()
{
(e => e.Name == "Ronald"),
(e => e.Age == 43),
(e => e.EyeColor == "Blue")
};
Executing the filters
You're in luck. Because you want to apply all the filters (logical AND), you can do this very easily by stacking them:
var myFilteredData = myContext.Set<Person>()
.Where(filter1)
.Where(filter2)
.Where(filter3)
.ToList();
Or, if you're using a List<Func<Person,bool>>:
var myFilteredData = myContext.Set<Person>().AsQueryable();
foreach(var filter in listOfFilters)
{
myFilteredData = myFilteredData.Where(filter);
}
Fringe cases
However, if you were trying to look for all items which fit one or more filters (logical OR), it becomes slightly more difficult.
The full answer is quite complicated.. You can check it here.
However, assuming you have the filters set in known variables, there is a simpler method:
Func<Person, bool> filterCombined =
e => filter1(e) || filter2(e) || filter3(e);
var myFilteredData = myContext.Set<Person>()
.Where(filterCombined)
.ToList();
One of the problems is that the collection of IQueryable is only valid as long as your DbSet is valid. As soon as your DbContext is Disposed your carefully filled collection is worthless.
So you have to think of another method to reconstruct the query than the one that uses the DbSet<Person>
Although at first glance they seem the same, there is a difference between IEnumerable and IQueryable. An Enumerable has everything in it to enumerate over the resulting sequence.
A Queryable on the other hand holds an Expression and a Provider. The Provider knows where the data can be fetched. This is usually a database, but it can also be a CSV-file or other items where you can fetch sequences. It is the task of the Provider to interpret the Expression and to translate it info a format that the database can understand, usually SQL.
While concatenating the linq statements into one big linq statements, the database is not accessed. Only the Expression is changed.
Once you call GetEnumerator() and MoveNext() (usually by doing ForEach, or ToList(), or similar), the Expression is sent to the Provider who will translate it into a query format that the database understands and perform the query. the result of the query is an Enumerable sequence, so Getenumerator() and MoveNext() of the provider's query result are called.
Because your IQueryable holds this Provider, you can't enumerate anymore after the Provider has been disposed.
When using entity framework, the DbSet holds the Provider. In the Provider is the Database information held by the DbContext. Once you Dispose the DbContext you can't use the IQueryable anymore:
IQueryable<Person> query = null;
using (var dbContext = new MyDbcontext())
{
query = dbContext.Persons.Where(person => person.Age > 20);
}
foreach (var person in query)
{
// expect exception: the DbContext is already Disposed
}
So you can't put the Provider in your collection or possible queries. However, you could remember the Expression. The only thing your require from your Expression is that it returns a Person. You also need a function that takes this Expression and a QueryaProvider for Persons to convert it to an IQueryable.
Let's create a generic function for this, so It can be used for any type, not just for Persons:
static IQueryable<TSource> ToQueryable<TSource>(this IQueryProvider provider,
Expression expression)
{
return provider.CreateQuery(expression);
}
// well, let's add the following also:
static IQueryable<Tsource> ToQueryable<TSource>(this DbContext dbContext,
Expression expression)
{
return dbContext.Set<TSource>.Provider.ToQueryable<TSource>(expression);
}
For help on extension functions see Extension Functions Demystified
Now only once you create your collection of Expressions. For fast lookup make it a Dictionary:
enum PersonQuery
{
ByFirstname,
ByAge,
ByHasChildWithAgeOver,
Skip,
}
public IReadOnlyDictionary<PersonQuery, Expression> CreateExpressions()
{
Dictionary<PersonQuery, Expression> dict = new Dictionary<PersonQuery, Expression>();
using (var dbContext = new MyDbContext())
{
IQueryable<Person> queryByFirstName = dbContext.Persons
.Where(...);
dict.Add(PersonQuery.ByfirstName, queryByFirstName.Expression);
... // etc for the other queries
}
return dict.
}
Usage:
IReadOnlyCollection<Person> PerformQuery(PersonQuery queryId)
{
using (var dbContext = new MyDbContext())
{
// get the Expression from the dictionary:
var expression = this.QueryDictionary[queryId];
// translate to IQueryable:
var query = dbContext.ToQueryable<Person>(expression);
// perform the query:
return query.ToList();
// because all items are fetched by now, the DbContext may be Disposed
}
}
I am new to Lambda expression. I want a result from combination of two tables with where clause in lambda Expression , query runs fine but how to get result in variable after processing query??
var Rental = db.AUCDATA_COUPONS.Join(db.AUCDATA_TENORS,
c => c.AUCDT_ID,
o => o.AUCDT_ID,
(c, o) => new { c, o })
.Where(x => x.o.PRODUCT_ID == "SUKUK" && x.o.ISSUE_DATE == Convert.ToDateTime("02-MAR-12") && x.o.TENOR_ID == "03Y"
&& x.c.AUCDT_ID == x.o.AUCDT_ID && x.c.COUPON_NXTDT == Convert.ToDateTime("21-NOV-15"))
.Select(x => x.c.RENTAL_RATE);
db.AUCDATA_COUPONS is an IQueryable<T> (where T is the type of the class representing the table). The extension methods you use (like Join, Where and Select) take this IQueryable<T> and return a new IQueryable<T>.
The last Select returns an IQueryable<int> (or double depending of the type of RENTAL_RATE). The actual query (the lambdas) are only executed when you iterate through that IQueryable. You can do that with foreach
foreach(var rentalRate in Rental)
Maybe a better way is to convert the result to a list or array. This way you would execute the query only once and not again and again with every foreach you execute:
var list = Rental.ToList(); // results in an List<int>
// or
var array = Rental.ToArray(); // results in an int[]
Note that you'll probably need to change your datetime comparisons to
x.o.ISSUE_DATE.Date == new DateTime(2012,3,2)
and
x.c.COUPON_NXTDT.Date == new DateTime(2015,11,21)
for the query to work correctly.
You already have the results in the variable. But, depending on what you want to do with them, you can add .ToArray() or .ToList() after .Select(...).
I have next chunk of code
var query = wordCollection.Select((word) => { return word.ToUpper(); })
.Where((word) =>
{
return String.IsNullOrEmpty(word);
})
.ToList();
Suppose I want to refactor this code and extract the lambda expression from Where clause. In Visual Studio I just select this lambda and do Refactor -> Extract Method. By doing so I have my LINQ modified to
var query = wordCollection.Select((word) => { return word.ToUpper(); })
.Where(NewMethod1())
.ToList();
and a NewMethod1() is declared as
private static Func<string, bool> NewMethod1()
{
return (word) =>
{
return String.IsNullOrEmpty(word);
};
}
The question is why this new method does NOT have any input parameters, as delegate Func states that NewMethod1() should have a string input parameter?
To get the expected result, mark just this part String.IsNullOrEmpty(word) and Extract the method:
private bool NewMethod(string word)
{
return String.IsNullOrEmpty(word);
}
What you originally got is because the extract created a method that returns a delegate. Not a method that matches the delegate. It is a method that returns another method. The latter accepts a string parameter word and returns a bool result.
Sure doing the above changes your code to:
.Where((word) => NewMethod(word))
But you can safely change that to:
.Where(NewMethod)
Side Note:
No need to use the return keyword in your Linq Queries or any one-line lambda, you can refactor you query to be like this:
var query = wordCollection.Select(word => word.ToUpper())
.Where(word => string.IsNullOrEmpty(word))
.ToList();
You are selecting the whole lambda, so it is trying to extract the whole lambda statement as a delegate that takes in a word and returns a boolean - Func < string, bool>.
When refactoring you should have only selected the "return String.IsNullOrEmpty(word);" part.
Additionally, you are using the lambas in an unnecessarily complex way.
You could refactor your LINQ statement to this:
var query = wordCollection.Select(word => word.ToUpper())
.Where(word => String.IsNullOrEmpty(word))
.ToList();
Or even to this:
var query = wordCollection.Select(word => word.ToUpper())
.Where(String.IsNullOrEmpty)
.ToList();
Can anyone tell me why these two code snippets give me two different SQL executions:
First
return nHibernateSession.Query<TEntity>()
.Where(_filter)
.Select(_selector)
.FirstOrDefault();
Where I pass in the arguments
Func<Product, bool> _filter = x => x.Id == 10;
Func<Product, string> _selector = x => x.Name;
When inspecting the query using NHibernate Profiler, this shows that I fully hydrate the entire collection of products in an unbounded query. It selects all fields and all rows, and then I guess it filters the resultset and only returns the name.
Second
And one where I have explicitly specified my query. I tried this to debug the previous generic expression.
return nHibernateSession.Query<Product>()
.Where(x => x.Id == 10)
.Select(x => x.Name)
.FirstOrDefault();
This one behaves as I would expect the previous one to behave. It only selects the name column, and applies a WHERE clause that ensures I only get product 10.
NHibernate needs the expression tree of the filter in order to generate SQL from it. When you pass a Func to it, NHibernate cannot translate it to SQL, so it retrieves all data from DB and then applies the Func filter to it.
change the first one to
Expression<Func<Product, bool>> _filter = x => x.Id == 10;
Expression<Func<Product, string>> _selector = x => x.Name;
In this case you give the whole expression to NHibernate, therefore a restricted query can be generated based on it.
I'm building a dynamic query that can have n of Where method calls and n of SelectMany calls dependent upon user input. For example I may have:
var qZ = entityContext.TableA
.SelectMany(a=>a.TableB, (a,t)=>new{a,t} )
.Where(a=>a.t.FieldID==21)
.Where(a=> EntityFunctions.Left(a.t.Value,1)=="p")
.SelectMany(a=>a.a.TableC, (a,t)=>new{a,t} )
.Where(a=>a.t.FieldID==22)
.Where(a=> a.a.t.Value=="Peter" && a.t.Value=="Pan")
.Where(a=> a.a.a.TypeID==3)
.Select(a=> new{ a.a.a.ItemID }
).Distinct();
In the method I'm writing, I use helper methods that return an IQueryable as seen in the return line below.
return query.Provider.CreateQuery(
Expression.Call(typeof(Queryable),
"Where",
new Type[] {query.ElementType},
query.Expression, predicateLambda)
);
I'm able to create LambdaExpressions for all of the various query attribute-value pairs required, but I am unable to create one for the resultSelector of Queryable.SelectMany.
How can we create (a,t) => new{a=a, t=t} in an expression tree? Or How do we accomplish the same result as the .SelectMany above using Expression.Call like below?
Expression.Call(typeof(Queryable),
"SelectMany",
????????,
????????
);
I've tried using the SelectMany overload that doesn't require the resultSelector which works to some degree, however, I don't know how to reference the properties of t in subsequent method calls.
I've found this lambda expression ((a,t) => new{a=a, t=t}) associated with SelectMany all over the web, but I can't find any example of how to convert it to an expression tree.
UPDATE:
Let's reframe the question. I can pass the lambda like this
var q = entityContext.TableA.AsQueryable();
var q1 = Queryable.SelectMany(q, a => a.TableB, (a, t) => new { a = a, t = t });
var q2 = Queryable.Where(q1,a=>a.t.FieldID==22);
That works, however, since I don't know ahead of time how many SelectMany need to be called and since each call changes to anonymous type of the IQueriable, is there a way to cast (and re-cast) the anonymous type to a single variable? This way I can loop through and apply whatever method necessary to the variable and then enumerate to get the results once the query is built. Something like:
var q = entityContext.TableA..AsQueryable();
q = Queryable.SelectMany(q, a => a.TableB, (a, t) => new { a = a, t = t });
q = Queryable.Where(q,a=>a.t.FieldID==22);
(BTW: This doesn't work)
The way that I ended up resolving this required a paradigm shift. The first query above was based upon the fact that I learned to write queries by joining all the tables I needed together to give me access to filter on and select fields in those tables.
SelectMany() creates joins and the box around my thinking at the time required that if I need to filter on a specific column in a table, I had to join that table to my query. This in turn changed the type of my IQueryable resulting in my not being able to predict the Type of the IQueryable at design time.
Answer:
Step 1: Set the type of IQueryable to the output type it needs to return. In the case above, the result was always IQueryable.
Step 2: Utilize Expressions to dynamically create the WHERE predicate, including any and all tables necessary to create the proper filter. This always returns Expression> and all of the other variables we easily accounted for. And rememeber, in EF it isn't necessary to join table outside of Where() if they are only needed in Where().