We Keep Coding

LINQ using String Format

I am getting the error about LINQ to Entities does not recognize the method 'System.String Format but in the past I was able to do this when I have included .AsEnumerable() is there something different I need to do because of the GroupBy section?
select new PresentationLayer.Models.PanelMeeting
{
GroupId = pg.GroupId,
MeetingId = pmd.MeetingId,
GuidelineName = pmv.GuidelineName,
PanelDisclosuresAttendanceURL = string.Format("{0}?MeetingId={1}&GroupId=0",PanelDisclosureLink, pmd.MeetingId),
}).GroupBy(g => new
{
g.MeetingId,
g.GroupId
})
.AsEnumerable()
.SelectMany(grp => grp.AsEnumerable()).ToList(),

You have to be aware of the difference between an IEnumerable<...> and an IQueryable<...>.
IEnumerable
An object that implements IEnumerable<...> represents a sequence of similar items. You can ask for the first element of the sequence, and as long as you've got elements you can ask for the next element. IEnumerable objects are supposed to be executed within your own process. IEnumerable objects hold everything to enumerate the sequence.
At its lowest level, this is done using GetEnumerator() / MoveNext() / Current:
IEnumerable<Customer> customers = ...
IEnumerator<Customer> enumerator = customers.GetEnumerator();
while (enumerator.MoveNext())
{
// There is a next Customer
Customer customer = enumerator.Current;
ProcessCustomer(customer);
}
If you use foreach, then internally GetEnumerator / MoveNext / Current are called.
If you look closely to LINQ, you will see that there are two groups of LINQ methods. Those that return IEnumerable<TResult> and those that dont't return IEnumerable<...>
LINQ functions from the first group won't enumerate the query. They use deferred execution, or lazy execution. In the comments section of every LINQ method, you'll find this description.
The LINQ functions of the other group will execute the query. If you look at the reference source of extension class Enumerable, you'll see that they internally use foreach, or at lower level use GetEnumerator / MoveNext / Current
IQueryable
An object that implements IQueryable<...> seems like an IEnumerable. However, it represents the potential to fetch data for an Enumerable sequence. The data is usually provided by a different process.
For this, the IQueryable holds an Expression and a Provider. The Expression represents what must be fetched in some generic format. The Provider knows who will provide the data (usually a database management system) and how to communicate with this DBMS (usually SQL).
When you start enumerating the sequence, deep inside using GetEnumerator, the Expression is sent to the Provider, who will try to translate it into SQL. The data is fetched from the DBMS, and returned as an Enumerable object. The fetched data is accessed by repeatedly calling MoveNext / Current.
Because the database is not contacted until you start enumerating, you'll have to keep the connection to the database open until you've finished enumerating. You've probably made the following mistake once:
IQueryable<Customer> customers;
using (var dbContext = new OrderDbContext(...))
{
customers = dbContext.Customers.Where(customer => customer...);
}
var fetchedCustomers = customers.ToList();
Back to your question
In your query, you use string.Format(...). Your Provider doesn't know how to translate this method into SQL. Your Provider also doesn't know any of your local methods. In fact, there are even several standard LINQ methods that are not supported by LINQ to entities. See Supported and Unsupported LINQ methods.
How to solve the problem?
If you need to call unsupported methods, you can use AsEnumerable to fetch the data. All LINQ methods after AsEnumerable are executed by your own process. Hence you can call any of your own functions.
Database Management systems are extremely optimized in table handling. One of the slower parts of a database query is the transport of the selected data to your local process. Hence, let the DBMS do all selecting, try to transport as little data as possible.
So let your DBMS do your Where / (Group-)Join / Sum / FirstOrDefault / Any etc. String formatting can be done best by you.
In your String.Format you use PanelDisclosureLink and pmd.MeetingId. It will probably be faster if your process does the formatting. Alas you forgot to give us the beginning or your query.
I'm not sure where your PanelDisclosureLink comes from. Is it a local variable? If that is the case, then PanelDisclosuresAttendanceURL will be the same string for every item in your group. Is this intended?
var panelDisclosureLine = ...;
var result = dbContext... // probably some joining with Pgs, Pmds and Pmvs,
.Select(... => new
{
GroupId = pg.GroupId,
MeetingId = pmd.MeetingId,
GuidelineName = pmv.GuidelineName,
})
// make groups with same combinations of [MeetingId, GroupId]
.GroupBy(joinResult => new
{
MeetingId = joinResult.MeetingId,
GroupId = joinResult.GroupId,
},
// parameter resultSelector: use the Key, and all JoinResult items that have this key
// to make one new:
(key, joinResultItemsWithThisKey) => new
{
MeetingId = key.MeetingId,
GroupId = key.GroupId,
GuideLineNames = joinResultsItemsWithThisKey
.Select(joinResultItem => joinResultItem.GuideLineName)
.ToList(),
})
So by now the DBMS has transformed your join result into objects with
[MeetingId, GroupId] combinations and a list of all GuideLineNames that have belong to
this [MeetingId, GroupId] combination.
Now you can move it to your local process and use String.Format.
.AsEnumerable()
.SelectMany (fetchedItem => fetchedItem.GuideLineNames,
(fetchedItem, guideLineName) => PresentationLayer.Models.PanelMeeting
{
GroupId = fetchedItem.GroupId,
MeetingId = fetchedItem.MeetingId,
GuidelineName = guidelineName,
PanelDisclosuresAttendanceURL = string.Format("...",
PanelDisclosureLink,
fetchedItem.MeetingId);
Note: in my parameter choice plurals are collections; singulars are elements of these collections.
PanelDisclosuresAttendanceURL = string.Format("{0}?MeetingId={1}&GroupId=0",PanelDisclosureLink, pmd.MeetingId),
}).
.GroupBy

If you want to use string.Format you first have to get the data from the server.
You can just move the .GroupBy( ... ) and then the .AsEnumerable() call to the top, before select new PresentationLayer.Models.PanelMeeting { ... }. If you are not selecting too much data that way...

Apply "List" of IQueryables to a target?

I have this idea to create a "list" of IQueryables that do different kinds of operations.
So basically:
var query1 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Name == "Ronald");
var query2 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Age == 43);
var query3 = Enumerable.Empty<Person>().AsQueryable().Select(e => e.EyeColor);
var listOfQueries = new List<IQueryable<Person>
{
query1,
query2,
query3
};
Now, I also have this DbSet full of "Persons" and I would like to "apply" all my queries against that DbSet. How would I do that? Is it even possible?
An updated example:
var personQueryFactory = new PersonQueryFactory();
var personQueryByFirstname = personQueryFactory.CreateQueryByFirstname("Ronald"); //Query by Firstname.
var personQueryByAge = personQueryFactory.CreateQueryByAge(42); //Query by age.
var personQueryByHasChildWithAgeOver = personQueryFactory.CreateQueryByChildAgeOver(25); //Query using a "join" to the child-relationship.
var personQuerySkip = personQueryFactory.Take(5); //Only get the 5 first matching the queries.
var personQuery = personQueryFactory.AggregateQueries //Aggragate all the queries into one single query.
(
personQueryByFirstname,
personQueryByAge,
personQueryByHasChildWithAgeOver,
personQuerySkip
);
var personSurnames = personsService.Query(personQuery, e => new { Surname = e.Surname }); //Get only the surname of the first 5 persons with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
var personDomainObjects = personsService.Query<DomainPerson>(personQuery); //Get the first 5 persons as a domain-object (mapping behind the "scenes") with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
var personDaos = personsService.Query(personQuery); //Get the first 5 persons as a DAO-objects/entityframework-entities with a firstname of "Ronald" with the age 42 and that has a child thats over 25 years old.
The reason for doing this would be to create a more "unified" way of creating and re-using predefined queries, and then to be able to execute them against the DbSet and return the result as a domain-object and not an "entity-framework-model/object"

This is possible, but you need to reframe your approach slightly.
Storing the filters
var query1 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Name == "Ronald");
var query2 = Enumerable.Empty<Person>().AsQueryable().Where(e => e.Age == 43);
var query3 = Enumerable.Empty<Person>().AsQueryable().Select(e => e.EyeColor);
Reading your intention, you don't actually want to handle IQueryable objects, but rather the parameter that you supply to the Where method.
Edit: I missed that the third one was a Select(), not a Where(). Ive adjusted the rest of the answer as if this had also been a Where(). See the comments below for my response as to why you can't mix Select() with Where() easily.
Func<Person,bool> filter1 = (e => e.Name == "Ronald");
Func<Person,bool> filter2 = (e => e.Age == 43);
Func<Person,bool> filter3 = (e => e.EyeColor == "Blue");
This stores the same information (filter criteria), but it doesn't wrap each filter in an IQueryable of its own.
A short explanation
Notice the Func<A,B> notation. In this case, A is the input type (Person), and B is the output type (bool).
This can be extended further. A Func<string,Person,bool> has two input parameters (string, Person) and one output parameter (bool). A usage example:
Func<string, Person, bool> filter = (inputString, inputPerson) => inputString == "TEST" && inputPerson.Age > 35;
There is always one output parameter (the last type). Every other mentioned type is an input parameter.
Putting the filters in a list
Intuitively, since Func<Person,bool> represents a single filter; you can represent a list of filters by using a List<Func<Person,bool>>.
Nesting generic types get a bit hard to read, but it does work just like any other List<T>.
List<Func<Person,bool>> listOfFilters = new List<Func<Person,bool>>()
{
(e => e.Name == "Ronald"),
(e => e.Age == 43),
(e => e.EyeColor == "Blue")
};
Executing the filters
You're in luck. Because you want to apply all the filters (logical AND), you can do this very easily by stacking them:
var myFilteredData = myContext.Set<Person>()
.Where(filter1)
.Where(filter2)
.Where(filter3)
.ToList();
Or, if you're using a List<Func<Person,bool>>:
var myFilteredData = myContext.Set<Person>().AsQueryable();
foreach(var filter in listOfFilters)
{
myFilteredData = myFilteredData.Where(filter);
}
Fringe cases
However, if you were trying to look for all items which fit one or more filters (logical OR), it becomes slightly more difficult.
The full answer is quite complicated.. You can check it here.
However, assuming you have the filters set in known variables, there is a simpler method:
Func<Person, bool> filterCombined =
e => filter1(e) || filter2(e) || filter3(e);
var myFilteredData = myContext.Set<Person>()
.Where(filterCombined)
.ToList();

One of the problems is that the collection of IQueryable is only valid as long as your DbSet is valid. As soon as your DbContext is Disposed your carefully filled collection is worthless.
So you have to think of another method to reconstruct the query than the one that uses the DbSet<Person>
Although at first glance they seem the same, there is a difference between IEnumerable and IQueryable. An Enumerable has everything in it to enumerate over the resulting sequence.
A Queryable on the other hand holds an Expression and a Provider. The Provider knows where the data can be fetched. This is usually a database, but it can also be a CSV-file or other items where you can fetch sequences. It is the task of the Provider to interpret the Expression and to translate it info a format that the database can understand, usually SQL.
While concatenating the linq statements into one big linq statements, the database is not accessed. Only the Expression is changed.
Once you call GetEnumerator() and MoveNext() (usually by doing ForEach, or ToList(), or similar), the Expression is sent to the Provider who will translate it into a query format that the database understands and perform the query. the result of the query is an Enumerable sequence, so Getenumerator() and MoveNext() of the provider's query result are called.
Because your IQueryable holds this Provider, you can't enumerate anymore after the Provider has been disposed.
When using entity framework, the DbSet holds the Provider. In the Provider is the Database information held by the DbContext. Once you Dispose the DbContext you can't use the IQueryable anymore:
IQueryable<Person> query = null;
using (var dbContext = new MyDbcontext())
{
query = dbContext.Persons.Where(person => person.Age > 20);
}
foreach (var person in query)
{
// expect exception: the DbContext is already Disposed
}
So you can't put the Provider in your collection or possible queries. However, you could remember the Expression. The only thing your require from your Expression is that it returns a Person. You also need a function that takes this Expression and a QueryaProvider for Persons to convert it to an IQueryable.
Let's create a generic function for this, so It can be used for any type, not just for Persons:
static IQueryable<TSource> ToQueryable<TSource>(this IQueryProvider provider,
Expression expression)
{
return provider.CreateQuery(expression);
}
// well, let's add the following also:
static IQueryable<Tsource> ToQueryable<TSource>(this DbContext dbContext,
Expression expression)
{
return dbContext.Set<TSource>.Provider.ToQueryable<TSource>(expression);
}
For help on extension functions see Extension Functions Demystified
Now only once you create your collection of Expressions. For fast lookup make it a Dictionary:
enum PersonQuery
{
ByFirstname,
ByAge,
ByHasChildWithAgeOver,
Skip,
}
public IReadOnlyDictionary<PersonQuery, Expression> CreateExpressions()
{
Dictionary<PersonQuery, Expression> dict = new Dictionary<PersonQuery, Expression>();
using (var dbContext = new MyDbContext())
{
IQueryable<Person> queryByFirstName = dbContext.Persons
.Where(...);
dict.Add(PersonQuery.ByfirstName, queryByFirstName.Expression);
... // etc for the other queries
}
return dict.
}
Usage:
IReadOnlyCollection<Person> PerformQuery(PersonQuery queryId)
{
using (var dbContext = new MyDbContext())
{
// get the Expression from the dictionary:
var expression = this.QueryDictionary[queryId];
// translate to IQueryable:
var query = dbContext.ToQueryable<Person>(expression);
// perform the query:
return query.ToList();
// because all items are fetched by now, the DbContext may be Disposed
}
}

Can't get deferred LINQ statements to work in nHibernate

Trying to write dynamic queries using the LINQ provider for NHibernate, but I am having issues. My understanding was that LINQ queries were deferred until called, (i.e. with ToList()), so I have the following code:
string[] filteredIds = new[] { "someIdNotInUse"};
var result = _products
.GetAll()
.Skip(0)
.Take(10);
if (filteredIds != null)
{
result.Where(x => x.Child1.Child2.Any(z => filteredIds.Contains(z.Child3.Id)));
}
var r = result.ToList();
The Where filter in the conditional block is not applied; when I run .ToList, I get records where I expect none. However, if I remove the where filter and append it directly to the _products call, it works as expected. Am I misunderstanding how the LINQ provider works? How is creating a query like this possible, without rewriting the query for every possible filter condition and combination?

Methods in LINQ don't affect the object they're called on - they return a new object representing the result of the call. So you want:
if (filteredIds != null)
{
result = result.Where(...);
}
(Think of it as being a bit like calling Replace or Trim on a string - the string is immutable, so it's only the return value which matters.)

Using Count with Take with LINQ

Is there a way to get the whole count when using the Take operator?

You can do both.
IEnumerable<T> query = ...complicated query;
int c = query.Count();
query = query.Take(n);
Just execute the count before the take. this will cause the query to be executed twice, but i believe that that is unavoidable.
if this is in a Linq2SQL context, as your comment implies then this will in fact query the database twice. As far as lazy loading goes though it will depend on how the result of the query is actually used.
For example: if you have two tables say Product and ProductVersion where each Product has multiple ProductVersions associated via a foreign key.
if this is your query:
var query = db.Products.Where(p => complicated condition).OrderBy(p => p.Name).ThenBy(...).Select(p => p);
where you are just selecting Products but after executing the query:
var results = query.ToList();//forces query execution
results[0].ProductVersions;//<-- Lazy loading occurs
if you reference any foreign key or related object that was not part of the original query then it will be lazy loaded in. In your case, the count will not cause any lazy loading because it is simply returning an int. but depending on what you actually do with the result of the Take() you may or may not have Lazy loading occur. Sometimes it can be difficult to tell if you have LazyLoading ocurring, to check you should log your queries using the DataContext.Log property.

The easiest way would be to just do a Count of the query, and then do Take:
var q = ...;
var count = q.Count();
var result = q.Take(...);

It is possible to do this in a single Linq-to-SQL query (where only one SQL statement will be executed). The generated SQL does look unpleasant though, so your performance may vary.
If this is your query:
IQueryable<Person> yourQuery = People
.Where(x => /* complicated query .. */);
You can append the following to it:
var result = yourQuery
.GroupBy (x => true) // This will match all of the rows from your query ..
.Select (g => new {
// .. so 'g', the group, will then contain all of the rows from your query.
CountAll = g.Count(),
TakeFive = g.Take(5),
// We could also query for a max value.
MaxAgeFromAll = g.Max(x => x.PersonAge)
})
.FirstOrDefault();
Which will let you access your data like so:
// Check that result is not null before access.
// If there are no records to find, then 'result' will return null (because of the grouping)
if(result != null) {
var count = result.CountAll;
var firstFiveRows = result.TakeFive;
var maxPersonAge = result.MaxAgeFromAll;
}

Lambda syntax in linq to db4o?

I know the following is possible with linq2db4o
from Apple a in db
where a.Color.Equals(Colors.Green)
select a
What I need however is something that allows me to build my query conditionally (like I can in other linq variants)
public IEnumerable<Apple> SearchApples (AppleSearchbag bag){
var q = db.Apples;
if(bag.Color != null){
q = q.Where(a=>a.Color.Equals(bag.Color));
}
return q.AsEnumerable();
}
In a real world situation the searchbag will hold many properties and building a giant if-tree that catches all possible combinations of filled in properties would be madman's work.
It is possible to first call
var q = (from Color c in db select c);
and then continue from there. but this is not exactly what I'm looking for.
Disclaimer: near duplicate of my question of nearly 11 months ago.
This one's a bit more clear as I understand the matter better now and I hope by now some of the db4o dev eyes could catch this on this:
Any suggestions?

Yes it's definitely possible to compose optimized LINQ queries using db4o. Granted that db is defined as follows:
IObjectContainer db;
Here is your query:
public IEnumerable<Apple> SearchApples (AppleSearchbag bag)
{
var query = db.Cast<Apple> ();
// query will be a Db4objects.Db4o.Linq.IDb4oLinqQuery<Apple>
if (bag.Color != null)
query = query.Where (a => a.Color == bag.Color);
return query;
}
In that case, the query will be executed whenever the returned enumerable is being iterated over.
Another possibility is to use the IQueryable mechanism, that has the advantage of being better recognized by developers:
public IQueryable<Apple> SearchApples (AppleSearchbag bag)
{
var query = db.AsQueryable<Apple> ();
// query will be a System.Linq.IQueryble<Apple>
if (bag.Color != null)
query = query.Where (a => a.Color == bag.Color);
return query;
}
In both cases, db4o will try to deduce an optimized query from the lambda expression upon execution, and if it fails, will fallback to LINQ to objects. The first one has the advantage of being more direct, by avoiding the queryable to LINQ to db4o transformation.