We Keep Coding

Using IQueryable with record type - InvalidOperationException

I am getting an InvalidOperationException when using a projection into a record type (.Net 7). The query works fine if the where clause is chained. It also works if where clause is not chained and the projection is not into a record (class, anonymous type both ok).
public record TestDto(
int CellLocationId,
string AreaCode
);
This works - Chained:
var query = _context.myTable.Where(x => x.AreaCode == areaCode).Select(s => new TestDto(s.CellLocationId, s.AreaCode));
This fails - Unchained with record:
var query = _context.myTable.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
query = query.Where(x => x.AreaCode == areaCode);
This works - Unchained with anonymous type:
var query = _context.myTable.Select(s => new {s.CellLocationId, s.AreaCode});
query = query.Where(x => x.AreaCode == areaCode);
When projecting into a record the sql shown in the error appears as:
System.InvalidOperationException: The LINQ expression 'DbSet().Where(v => new TestDto(v.CellLocationId, v.AreaCode).AreaCode == __areaCode_0)' could not be translated.
Why can I not use a record when my IQueryable is using an unchained where clause?
15/11/2022 Update.
Github issue
Possible workarounds - project into class, anonymous type or as per Bagus Tesa answer (and my preferred option) project into record type at end of statement.

About IEnumerable and IQueryable
IQueryable extends IEnumerable
IEnumerable most of the time works in memory, with some exceptions
IQueryable will be translated into SQL Query
Both IQueryable and IEnumerable execution are delayed until they are materialized (e.g. using ToList, AsEnumerable on IQueryable, etc.)
The API is designed this way for convenience in working with database queries. It allows (somewhat) interchangeable linq chains between the two. However, there are some caveats that need to be kept in mind.
IQueryable can only infer whats within its Expression. You can't for example do something like this:
public int GetAge(DateTime date)
{
...
}
var firstUserAge = _context.users.First().Select(q => GetDate(q.DoY));
It won't work, see QA. Linq is simply unable to translate the GetDate method into SQL Query.
The Problem
Let's take a look with the third case (anonymous class).
var query = _context.myTable
.Select(s => new {s.CellLocationId, s.AreaCode});
query = query.Where(x => x.AreaCode == areaCode);
Up to _context.MyTable the only known type is simply DbSet<myTable>. However, as it reach the Select(s => ...) it became aware of the anonymous type. Linq the consider that query should be an IQueryable<..__AnonymousType..>. Given the complete information is given to it. Linq can translate it into (roughly) sql query shown below and the result snuggly fits the anonymous type.
select s.cellLocationId, s.AreaCode
from myTable as s
where s.AreaCode = ....
Now, let's see what happens in the first case which surprisingly works.
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode))
.Where(x => x.AreaCode == areaCode);
Hang on a second! Did not I previously mentioned that you can't call a method on your program's side from within an IQueryable?
Yes, your code above won't work in the past, see discussion. You can find numerous examples of similar problem. You can also see examples on EF Core 7 documentation that they used anonymous types all the time. The microsoft's official tutorial on DTO also assigns properties manually one-by-one without using constructor.
However, EF Core 7 is pretty new. It is said to have better performance compared to earlier EF Core. The changes may involve reworks on how it materialize IQueryable. There are plenty of issues need to be sorted out.
You should run query intercepts and see the queries being feed into the database to see the difference between the two cases above. I'd bet that the first case actually fetch everything in myTable.
What We Should Do
We can simply map into DTO at a later part of the code (at the very end of the chain).
var query = _context.MyTable;
if(someFlag)
{
query = query.Where(s => s.AreaCode = areaCode);
}
//... a bunch more filters
return query.AsEnumerable()
.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
This way, you can avoid the bug altogether while keeping most of the heavy-lifting on the database.
You may complain "I will have duplicate code for computing AreaCode." You have to remember, DTO in the first place does not contain business logic.

Regarding what Bagus Tesa wrote, the most important parts there are:
However, EF Core 7 is pretty new. It is said to have better performance compared to earlier EF Core. The changes may involve reworks on how it materialize IQueryable. There are plenty of issues need to be sorted out.
and
We can simply map into DTO at a later part of the code (at the very end of the chain).
I do not like how that answer is written, and I started writing my own, and then, frankly, I ended up with my own wall of text, and I don't like it just the same as their answer. Oh well. Maybe someone finds it useful. I mark it with community-wiki, because I don't really want to compete. It's more like, IDK, addendum?
Regarding the latter point, there have been some issues with early-selects since the dawn of EF. Select-as-custom-class is a special construct designed to allow your query to return your custom entity types (instead of anon bags of fields), especially in cases where you Join several tables, where the result cannot be simply the context.table you start the query from.
Thanks to how IQueryable is designed, you can moreless put such a .Select anywhere in your query, but there's a catch: once IQueryable is now <of-you-custom-type>, your custom properties are exposed instead of original tables columns, and it gets increasingly harder to guarantee that your code doesn't try to map/execute something untranslatable.
For example, imagine:
// case1:
var result = context.myTable
.Where(x => x.AreaCode == areaCode)
.Select<MyClass>(...)
.ToList();
// case2:
var result = context.myTable
.Select<MyClass>(...)
.Where(x => x.AreaCode == areaCode)
.ToList();
In the first case, it's obvious. AreaCode is a column from MyTable, defined by whatever type of MyTable.
In second case, it's AreaCode but from MyClass, and since someone wanted to introduce a new type, we can assume it's not mapped in EF mappings to the same table as MyTable. So if EF hanles that, it may have to so some guessing what MyClass.AreaCode really is.
Furthermore, what if MyClass.AreaCode was:
private string theCode = "5";
public string AreaCode {
set { theCode = value; }
get { return theCode + "1"; }
}
In that case, case1 would still function properly. And case2 can never function properly. Not even mentioning, custom class constructors, we write .Select(.. => new MyClas(...)) and it's very tempting to hide some custom logic into the ctor, even if it's as simple as if/throw/ ArgumentNullException.
If EF handles such cases it in ANY way (like: assume props/ctors are transparent, ignoring logic inside), results will be unintuitive to the programmer: MyClass used in Query will behave differently than MyClass used in the runtime. EF shouldn't handle it, it should throw, but determining that a getter or constructor is pass-through without custom logic isn't that easy.
That's why anon-types new {foo=, bar=} are always preferred (2) as the intermediate types - they are 100% compiler-generated, the compiler knows everything about them, the compiler knows all fields/properties are plain and transparent, and that's why it's not a huge problem if you put .Select<anon> even early in the LINQ query. (1)
But let's get back to your code.
In that record-related code samples you provided:
// first:
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode))
.Where(x => x.AreaCode == areaCode);
// second:
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
query = query.Where(x => x.AreaCode == areaCode);
These pieces of code, if there is really nothing else between context:get_myTable and IQueryable<record>:Where, something that you'd for some reason omit, then these two pieces of code are essentially the same. Or should be.
When we write return " mom ".ToUpper().Trim(); we don't expect it to have different results than if we write var x = " mom ".ToUpper(); x = x.Trim(); return x;, right? That's all complete basics of operation/result/variable semantics.
The only difference here is splitting such unmaterialized IQueryable 'result' of a .Select() to a variable. LINQ call chain is exactly the same. Names are exactly the same. Difference is purely syntactical. So if you see that one of them works, and the other fails, this means there's a bug in the compiler, and you should:
file a bug issue at roslyn github
get away from whatever this bug can be related to as far as possible (meaning not use records yet, or at least not early in the query)
or, at least test the hell out of it, so you know how to use it safely even bugged, and get as much of those tests automated
There's really not much else to be advised here.
(1) actually sometimes it is problematic. Every ".Select" in the chain changes the "shape of the query", as they call it sometimes in MSDN articles, and it usually breaks any .Include setups that were done on the original shape. But YMMV there.
(2) anon-types are perfect here from the LINQ point of view, but are often a nightmare for the developer who wants to do anything a bit more reusable or structured. You cannot return them easily from your methods, since the type is anonymous. And so on. This severely limits reuse, and I can easily imagine, that's why RecordTypes are introduced - to provide a plain type with guaranteed mechanics, a type that can be named and easily referred to.

Pass a lambda parameter to an include statement

I am using the System.Data.Entity namespace, so I can pass lambda expressions to the Linq Include method.
public ICollection<MyEntity> FindAll()
{
using (var ctx = new MyEntityContext())
{
return ctx.MyEntity.Include(x => x.SomeLazyLoadedValue).ToList();
}
}
When I'm using a Where statement in a different method, I can pass a parameter to it like so:
public ICollection<MyEntity> FindAllBy(Func<MyEntity, bool> criteria)
{
using (var ctx = new MyEntityContext())
{
return ctx.MyEntity.Where(criteria).ToList();
}
}
However, trying the same thing in an Include does not work:
public ICollection<MyEntity> FindAll(Func<MyEntity, bool> criteria)
{
using (var ctx = new MyEntityContext())
{
return ctx.MyEntity.Include(criteria).ToList();
}
}
If you try this, Visual Studio will complain that it
Cannot convert from 'System.Func<MyEntity, bool>' to 'string'
How do I pass a lambda to the Include method?

There are a few problems with your code. For instance, your FindAllBy does not do a sql WHERE query, instead it loads all the entries in your database, and then filter in-memory based on your criteria. To understand why this is like so take a look at the following:
int a = 5;
long b = 5;
Now, it's quite obvious what's happening here, but it's still quite important. The compiler reads the following code and produces two variables. One integer and one long integer, both with values set to the number 5. However, the values of these two numbers are different, even though they are set (in the source code) to the same thing. One is 32-bit, and the other is 64-bit.
Now, let's take a look at the following code:
Func<int, string> a = num => num.ToString();
Expr<Func<int, string>> b = num => num.ToString();
Here the same thing (more or less) is happening. In the first case, the C# compiler sees you want a predicate (a Func<int, string> predicate), whereas the second value is a Expr<Func<int, string>> even though the values are written the same. However, as opposed to the first example, the end result here is vastly different.
A predicate is compiled as a method on a compiler-generated class. It's compiled just as any other code, and simply allows you to remove a bunch of boilerplate. A expression on the other hand is a in-memory representation of the actual code written. In this case, for instance, the expression might look something akin to Call(int.ToString, $1). This can be read by other code and translated to for instance SQL which is then used to query your database.
Now, back to your problem. EntityFramework hands you IQueryable<T> instances, which in turn inherit IEnumerable<T>. Whenever you enumerate over the enumerable, it queries the database.
All the extension-methods that accept delegates are defined on IEnumerable and thus query your database before running the predicate. This is why you need to make sure to select the right method-overloads.
Edit (to answer comment)]
To clarify a bit more I'm going to make a few examples. Say for instance that we have a User class that cointains FirstName, LastName and Age, and the db collection is simply called db.
Expr<Func<User, bool>> olderThan10 = u => u.Age > 10;
Func<User, bool> youngerThan90 = u => u.Age < 90;
var users = db.Where(olderThan10).Where(youngerThan90);
This would result in SQL that finds all users that are older than 10, after which it would in-memory filter away everyone that was older than or equal to 90.
So passing a Func doesn't necessarily mean it queries the whole database. It just means it stops building on the query at that point, and executes it.
As for the next question, Expression<Func<T,bool>> is not a universal answer. It means "a expression that takes a T and returns a bool". In some cases, like .Include which started this whole question, you don't want to return a bool. You want to return whatever you want to include. So for instance, if we go back to our example of users, and amend a Father property on the user class which references another user, and we want to include it, in regular code we'd do
db.Include(u => u.Father);
Now. Here, u is a User, and the return value u.Father is also a user, so in this case u => u.Father is Expr<Func<User, User>> or Expr<Func<User, object>> (I don't know if entity-framework .Include accepts generic values or simply objects).
So your FindAll function should probably look like this:
public ICollection<TData> FindAll<TInclude>(Expr<Func<TData, TInclude>> include) {
using (var ctx = new TContext()) {
return ctx.T.Include(include).ToList();
}
}
Though, to be honest, this is pretty weird looking code, and it's likely that you're doing something else weird with your models given that you've (for instance) named them T and TContext. My guess is that you need to read up a bit on how generics works in C#.

Linq lambda not working but delegate does

OK, this is doing my head in - I'm not even sure how to search for this.
Here is the first part of my function:
var rules = context.Rules.Include(r => r.CreatedBy).Include(r => r.ModifiedBy);
IUserManager um = GetUserManager();
var currentUser = um.GetCurrent();
Can someone tell me why this works:
return rules.Where(delegate(Rule r)
{
return r.CreatedBy.CompanyID == currentUser.CompanyID;
});
but this doesn't:
return rules.Where(r => r.CreatedBy.CompanyID == currentUser.CompanyID);
It's EF Code first and CreatedBy and ModifiedBy are both virtual properties.
r.CreatedBy and currentUser are both instances of the same class (if you didn't already work this out)
What the second snippet is returning is an empty list. It's as though the eager loading isn't working and the lambda doesn't cause a lazy load.
Note: I've just discovered that if I change the first line to
var rules = context.Rules.Include(r => r.CreatedBy)
.Include(r => r.ModifiedBy).ToList()
then the lambda works. The question still stands though. Why to I have to use the ToList() or the delegate. I'm doing the same thing elsewhere in the same class and it works as I'd expect.
Thanks

This is because lambdas can be implicitly converted to delegates or to expression trees. In the first case, the delegate is converted to an expression tree because rules is IQueryable<> and overload resolution chooses Queryable.Where. When you use the anonymous function, however, that can't be converted to an expression tree, so overload resolution has to choose Enumerable.Where.
When you make rules into a List<>, that forces overload resolution to choose Enumerable.Where, because List<> does not implement IQueryable<>. You could use AsEnumerable() to achieve the same effect without the overhead of creating the list.
As to why this doesn't work when you're "doing the same thing elsewhere in the same class and it works as I'd expect," we might be able to help if you give an example of code that does work.

Returning multiple streams from LINQ query

I want to write a LINQ query which returns two streams of objects. In F# I would write a Seq expression which creates an IEnumerable of 2-tuples and then run Seq.unzip. What is the proper mechanism to do this in C# (on .NET 3.5)?
Cheers, Jurgen

Your best bet is probably to create a Pair<T1, T2> type and return a sequence of that. (Or use an anonymous type to do the same thing.)
You can then "unzip" it with:
var firstElements = pairs.Select(pair => pair.First);
var secondElements = pairs.Select(pair => pair.Second);
It's probably worth materializing pairs first though (e.g. call ToList() at the end of your first query) to avoid evaluating the query twice.
Basically this is exactly the same as your F# approach, but with no built-in support.

Due to the lack of tuples in C# you may create an anonymous type.
Semantics for this are:
someEnumerable.Select( inst => new { AnonTypeFirstStream = inst.FieldA, AnonTypeSecondStream = inst.FieldB });
This way you're not bound in the amount of streams you return, you can just add a field to the anonymous type pretty like you can add an element to a tuple.

Adding functonality to Linq-to-SQL objects to perform common selections

In a previous question I asked how to make "Computed properties" in a linq to sql object. The answer supplied there was sufficient for that specific case but now I've hit a similar snag in another case.
I have a database with Items that have to pass through a number of Steps. I want to have a function in my database that retrieves the Current step of the item that I can then build on. For example:
var x = db.Items.Where(item => item.Steps.CurrentStep().Completed == null);
The code to get the current step is:
Steps.OrderByDescending(step => step.Created).First();
So I tried to add an extension method to the EntitySet<Step> that returned a single Step like so:
public static OrderFlowItemStep CurrentStep(this EntitySet<OrderFlowItemStep> steps)
{
return steps.OrderByDescending(o => o.Created).First();
}
But when I try to execute the query at the top I get an error saying that the CurrentStep() function has no translation to SQL. Is there a way to add this functionality to Linq-to-SQL in any way or do I have to manually write the query every time? I tried to write the entire query out first but it's very long and if I ever change the way to get the active step of an item I have to go over all the code again.
I'm guessing that the CurrentStep() method has to return a Linq expression of some kind but I'm stuck as to how to implement it.

The problem is that CurrentStep is a normal method. Hence, the Expression contains a call to that method, and naturally SQL cannot execute arbitrary .NET methods.
You will need to represent the code as an Expression. I have one in depth example here: http://www.atrevido.net/blog/2007/09/06/Complicated+Functions+In+LINQ+To+SQL.aspx
Unfortunately, the C# 3.0 compiler has a huge omission and you cannot generate calls to Expressions. (i.e., you can't write "x => MyExpression(x)"). Working around it either requires you to write the Expression manually, or to use a delegate as a placeholder. Jomo Fisher has an interesting post about manipulating Expression trees in general.
Without actually having done it, the way I'd probably approach it is by making the CurrentStep function take the predicate you want to add ("Completed == null"). Then you can create a full Expression> predicate to hand off to Where. I'm lazy, so I'm going to do an example using String and Char (String contains Chars, just like Item contains Steps):
using System;
using System.Linq;
using System.Linq.Expressions;
class Program {
static void Main(string[] args) {
Console.WriteLine(StringPredicate(c => Char.IsDigit(c)));
var func = StringPredicate(c => Char.IsDigit(c)).Compile();
Console.WriteLine(func("h2ello"));
Console.WriteLine(func("2ello"));
}
public static Expression<Func<string,bool>> StringPredicate(Expression<Func<char,bool>> pred) {
Expression<Func<string, char>> get = s => s.First();
var p = Expression.Parameter(typeof(string), "s");
return Expression.Lambda<Func<string, bool>>(
Expression.Invoke(pred, Expression.Invoke(get, p)),
p);
}
}
So "func" is created by using StringPredicate to create an Expression. For the example, we compile it to execute it locally. In your case, you'd pass the whole predicate to "Where" so it gets translated to SQL.
The "get" expression is where you put your "extension" stuff (OrderByWhatever, First, etc.). This is then passed in to the predicate that's given to you.
Don't worry if it looks complicated; it sorta is at first. If you haven't done this kinda stuff before, it'll take a bit of time (the first time I did this kinda stuff, it took hours to get it right :|.. now it comes slightly easier). Also, as I mentioned, you can write a helper method to do this re-writing for you (so you don't directly need to use the Expression.Whatever methods), but I haven't seen any examples and haven't really needed it yet.

Check out my answer to "switch statement in linq" and see if that points you in the right direction...
The technique i demonstrate there is the one that got me past the scary "no translation to SQL" error.