We Keep Coding

'AsyncEnumerableReader' reached the configured maximum size, but not using AsyncEnumerable

I'm using EF Core 3.1.1 (dotnet core 3.1.1). And I want to return a large number of Car entities. Unfortunately I get the following error message:
'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.Internal.InternalDbSet`...
I know that there is another answered question regarding the same error. But I'm not doing an explicit async operation.
[HttpGet]
[ProducesResponseType(200, Type = typeof(Car[]))]
public IActionResult Index()
{
return Ok(_carsDataModelContext.Cars.AsEnumerable());
}
The _carDataModelContext.Car is just a simple entity that maps 1-on-1 to a table in the database. public virtual DbSet<Car> Cars { get; set; }
Originally I return Ok(_carsDataModelContext.Cars.AsQueryable()) because we need to support OData. But to be sure it wasn't OData that is messing things up I tried to return AsEnumerable, and remove the "[EnableQuery]" attribute from the method. But that still ends in the same error.
The only way to fix this, is if I return Ok(_carsDataModelContext.Cars.ToList())

All Ef Core IQueryable<T> implementations (DbSet<T>, EntityQueryable<T>) also implement the standard IAsyncEnumerable<T> interface (when used from .NET Core 3), so AsEnumerable(), AsQueryable() and AsAsyncEnumerable() simply return the same instance cast to the corresponding interface.
You can easily verify that with the following snippet:
var queryable = _carsDataModelContext.Cars.AsQueryable();
var enumerable = queryable.AsEnumerable();
var asyncEnumerable = queryable.AsAsyncEnumerable();
Debug.Assert(queryable == enumerable && queryable == asyncEnumerable);
So even though you are not returning explicitly IAsyncEnumerable<T>, the underlying object implements it and can be queried for. Knowing that Asp.Net Core is naturally async framework, we can safely assume that it checks if the object implements the new standard IAsyncEnumerable<T>, and uses that behind the scenes instead of IEnumerable<T>.
Of course when you use ToList(), the returned List<T> class does not implement IAsyncEnumerable<T>, hence the only option is to use IEnumerable<T>.
This should explain the 3.1 behavior. Note that before 3.0 there was no standard IAsyncEnumerable<T> interface. EF Core was implementing and returning its own async interface, but the .Net Core infrastructure was unaware of it, thus was unable to use it on behalf of you.
The only way to force the previous behavior without using ToList() / ToArray() and similar is to hide the underlying source (hence the IAsyncEnumerable<T>).
For IEnumerable<T> it's quite easy. All you need is to create custom extension method which uses C# iterator, e.g:
public static partial class Extensions
{
public static IEnumerable<T> ToEnumerable<T>(this IEnumerable<T> source)
{
foreach (var item in source)
yield return item;
}
}
and then use
return Ok(_carsDataModelContext.Cars.ToEnumerable());
If you want to return IQueryable<T>, the things get harder. Creating custom IQueryable<T> wrapper is not enough, you have to create custom IQueryProvider wrapper to make sure composing over returned wrapped IQueryable<T> would continue returning wrappers until the final IEnumerator<T> (or IEnumerator) is requested, and the returned underlying async enumerable is hidden with the aforementioned method.
Here is a simplified implementation of the above:
public static partial class Extensions
{
public static IQueryable<T> ToQueryable<T>(this IQueryable<T> source)
=> new Queryable<T>(new QueryProvider(source.Provider), source.Expression);
class Queryable<T> : IQueryable<T>
{
internal Queryable(IQueryProvider provider, Expression expression)
{
Provider = provider;
Expression = expression;
}
public Type ElementType => typeof(T);
public Expression Expression { get; }
public IQueryProvider Provider { get; }
public IEnumerator<T> GetEnumerator() => Provider.Execute<IEnumerable<T>>(Expression)
.ToEnumerable().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
class QueryProvider : IQueryProvider
{
private readonly IQueryProvider source;
internal QueryProvider(IQueryProvider source) => this.source = source;
public IQueryable CreateQuery(Expression expression)
{
var query = source.CreateQuery(expression);
return (IQueryable)Activator.CreateInstance(
typeof(Queryable<>).MakeGenericType(query.ElementType),
this, query.Expression);
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
=> new Queryable<TElement>(this, expression);
public object Execute(Expression expression) => source.Execute(expression);
public TResult Execute<TResult>(Expression expression) => source.Execute<TResult>(expression);
}
}
The query provider implementation is not fully correct, because it assumes that only the custom Queryable<T> will call Execute methods for creating IEnumerable<T>, and external calls will be used only for immediate methods like Count, FirstOrDefault, Max etc., but it should work for this scenario.
Other drawback of this implementation is that all EF Core specific Queryable extensions won't work, which might be an issue/showstopper if OData $expand relies on methods like Include / ThenInclude. But fixing that requires more complex implementation digging into EF Core internals.
With that being said, the usage of course would be:
return Ok(_carsDataModelContext.Cars.ToQueryable());

LINQ expression with generic class properties

I would like to pass an IQueryable and an array of ids to a method which filters the IQueryable based on those ids.
As the ids can be either long's or int's it should be solved generically.
I came up with the following:
public static IEnumerable<T> GetModified<TId, T>(IQueryable<T> objects, TId[] ids) where T : class
{
return objects.Where(j => ids.Contains((TId)j.GetType().GetProperty("Id").GetValue(j)));
}
Unfortunately I'm getting the exception:
LINQ to Entities does not recognize the method 'System.Object GetValue(System.Object)' method, and this method cannot be translated into a store expression.

The exception is normal, as getting properties through reflection is something that clearly cannot be translated to SQL.
One thing I would try is to create a generic interface that exposes an Id property of a given type:
public interface HasId<T> {
T Id { get; set; }
}
Now you could declare your entity as implementing HasId<int>, for example, if the Id was of type int.
The next step is to modify your method like so:
public static IEnumerable<T> GetModified<TId, T>
(IQueryable<T> objects, TId[] ids) where T : class, HasId<TId>
{
return objects.Where(j => ids.Contains(j.Id));
}
Note the added generic restriction: where T : class, HasId<TId>. This enables you to write the simplified j.Id, which returns a TId value, instead of resorting to reflection.
Please note that I haven't run or tested this code; it's just an idea that I got when I saw your problem and I hope it helps.
Update:
Here's another possible solution that doesn't require that you declare interfaces or change your classes in any way:
public static IEnumerable<T> GetModified<TId, T>
(IQueryable<T> objects, TId[] ids, Expression<Func<T, TId>> idSelector)
where T : class
{
return objects.Where(j => ids.Contains(idSelector(j)));
}
What I've done here is add the Expression<Func<T, TId>> idSelector parameter, an expression that can return the Id of a given instance of T.
You would call the method like that:
var modified = GetModified(dbObjects, yourIdArray, entity => entity.Id);
(only the third parameter being new; keep the others as you have them now).
Again, I haven't tested if this works or even compiles, as I don't have a computer with VS here :(.

Entity Framework doesn't support some of the .NET methods such as GetValue() since it does not translate to SQL (which is the code actually executed to the IQueryable. Try calling ToList to get the CLR object before doing reflection:
public static IEnumerable<T> GetModified<TId, T>(IQueryable<T> objects, TId[] ids) where T : class
{
return objects.ToList().Where(j => ids.Contains((TId)j.GetType().GetProperty("Id").GetValue(j)));
}

Generic query method in Entity Framework

In my DB I have tables who have an attribute int DeleteState. I want a generic method to query those tables. In other words a method who does this: Context.Table.Where(x => x.DeleteState == 0).
I thought I could do this:
public static class Extensions
{
public static IQueryable<T> Exists<T>(this IQueryable<T> qry) where T : IDeletable
{
return qry.Where(x => x.DeleteState == 0);
}
}
Where IDeletable is this:
public interface IDeletable
{
int DeleteState { get; set; }
}
Now I only have to add the IDeletable in the EF model:
public partial class Table : EntityObject, IDeletable { ... }
I did this with the templating mechanism.
Unfortunately, it doesn't work :( It compiles fine, but throws at runtime:
Unable to cast the type 'Table' to type 'IDeletable'. LINQ to Entities only supports casting Entity Data Model primitive types
if I call it like that:
Context.Table.Exists();
How can I solve this problem? Could you think of a fix or a different method to achieve similar results? Thx

The problem you have is that the Entity Framework can only work with an Expression Tree. Your function executes a query directly instead of building an Expression Tree.
A simpler solution would be to add a Model Defined Function.
A model defined function can be called directly on an instance of your context.

Maybe:
public static IQueryable<T> Exists<T>(this IQueryable<T> qry)
{
return qry.Where(x => (!typeof(IDeletable).IsAssignableFrom(x.GetType()) || typeof(IDeletable).IsAssignableFrom(x.GetType()) && ((IDeletable)x).DeleteState == 0));
}

Tsss, this is the answer: Linq Entity Framework generic filter method
I forgot about the class here:
... where T : class, IDeletable

Have you tried converting your objects to IDeletable before you actually query? e.g.
public static IQueryable<T> Exists<T>(this IQueryable<T> qry)
{
return qry.Select<T, IDeletable>(x => x).Where(x => x.DeleteState == 0).Cast<T>();
}
I haven't tested this code, however, the error rings a bell and I remember I had to do something similar.

Linq-to-entities, Generics and Precompiled Queries

I'm experimenting with linq and generics. For now, I just implemented a GetAll method which returns all records of the given type.
class BaseBL<T> where T : class
{
public IList<T> GetAll()
{
using (TestObjectContext entities = new TestObjectContext(...))
{
var result = from obj in entities.CreateObjectSet<T>() select obj;
return result.ToList();
}
}
}
This works fine. Next, I would like to precompile the query:
class BaseBL<T> where T : class
{
private readonly Func<ObjectContext, IQueryable<T>> cqGetAll =
CompiledQuery.Compile<ObjectContext, IQueryable<T>>(
(ctx) => from obj in ctx.CreateObjectSet<T>() select obj);
public IList<T> GetAll()
{
using (TestObjectContext entities = new TestObjectContext(...))
{
var result = cqGetAll.Invoke(entities);
return result.ToList();
}
}
}
Here, i get the following:
base {System.Exception} = {"LINQ to Entities does not recognize the method
'System.Data.Objects.ObjectSet`1[admin_model.TestEntity] CreateObjectSet[TestEntity]()'
method, and this method cannot be translated into a store expression."}
What is the problem with this? I guess the problem is with the result of the execution of the precompiled query, but I am unable to fanthom why.

I had this exception when I used methods inside the LINQ query that are not part of the entity model. The problem is that the precompiled query can't invoke the CreateObjectSet for the type TestEntity because the precompiled query is not part of the context that is used to invoke it.

Accessing properties through Generic type parameter

I'm trying to create a generic repository for my models. Currently i've 3 different models which have no relationship between them. (Contacts, Notes, Reminders).
class Repository<T> where T:class
{
public IQueryable<T> SearchExact(string keyword)
{
//Is there a way i can make the below line generic
//return db.ContactModels.Where(i => i.Name == keyword)
//I also tried db.GetTable<T>().Where(i => i.Name == keyword)
//But the variable i doesn't have the Name property since it would know it only in the runtime
//db also has a method ITable GetTable(Type modelType) but don't think if that would help me
}
}
In MainViewModel, I call the Search method like this:
Repository<ContactModel> _contactRepository = new Repository<ContactModel>();
public void Search(string keyword)
{
var filteredList = _contactRepository.SearchExact(keyword).ToList();
}
Solution:
Finally went with Ray's Dynamic Expression solution:
public IQueryable<TModel> SearchExact(string searchKeyword, string columnName)
{
ParameterExpression param = Expression.Parameter(typeof(TModel), "i");
Expression left = Expression.Property(param, typeof(TModel).GetProperty(columnName));
Expression right = Expression.Constant(searchKeyword);
Expression expr = Expression.Equal(left, right);
}
query = db.GetTable<TModel>().Where(Expression.Lambda<Func<TModel, bool>>(expr, param));

Interface solution
If you can add an interface to your object you can use that. For example you could define:
public interface IName
{
string Name { get; }
}
Then your repository could be declared as:
class Repository<T> where T:class, IName
{
public IQueryable<T> SearchExact(string keyword)
{
return db.GetTable<T>().Where(i => i.Name == keyword);
}
}
Alternate interface solution
Alternatively you could put the "where" on your SearchExact method by using a second generic parameter:
class Repository<T> where T:class
{
public IQueryable<T> SearchExact<U>(string keyword) where U: T,IName
{
return db.GetTable<U>().Where(i => i.Name == keyword);
}
}
This allows the Repository class to be used with objects that don't implement IName, whereas the SearchExact method can only be used with objects that implement IName.
Reflection solution
If you can't add an IName-like interface to your objects, you can use reflection instead:
class Repository<T> where T:class
{
static PropertyInfo _nameProperty = typeof(T).GetProperty("Name");
public IQueryable<T> SearchExact(string keyword)
{
return db.GetTable<T>().Where(i => (string)_nameProperty.GetValue(i) == keyword);
}
}
This is slower than using an interface, but sometimes it is the only way.
More notes on interface solution and why you might use it
In your comment you mention that you can't use an interface but don't explain why. You say "Nothing in common is present in the three models. So i think making an interface out of them is not possible." From your question I understood that all three models have a "Name" property. In that case, it is possible to implement an interface on all three. Just implement the interface as shown and ", IName" to each of your three class definitions. This will give you the best performance for both local queries and SQL generation.
Even if the properties in question are not all called "Name", you can still use the nterface solution by adding a "Name" property to each and having its getter and setter access the other property.
Expression solution
If the IName solution won't work and you need the SQL conversion to work, you can do this by building your LINQ query using Expressions. This more work and is significantly less efficient for local use but will convert to SQL well. The code would be something like this:
class Repository<T> where T:Class
{
public IQueryable<T> SearchExact(string keyword,
Expression<Func<T,string>> getNameExpression)
{
var param = Expression.Parameter(typeof(T), "i");
return db.GetTable<T>().Where(
Expression.Lambda<Func<T,bool>>(
Expression.Equal(
Expression.Invoke(
Expression.Constant(getNameExpression),
param),
Expression.Constant(keyword),
param));
}
}
and it would be called thusly:
repository.SearchExact("Text To Find", i => i.Name)

Ray's method is quite good, and if you have the ability to add an interface definitely the superior however if for some reason you are unable to add an interface to these classes (Part of a class library you can't edit or something) then you could also consider passing a Func in which could tell it how to get the name.
EG:
class Repository<T>
{
public IQueryable<T> SearchExact(string keyword, Func<T, string> getSearchField)
{
return db.GetTable<T>().Where(i => getSearchField(i) == keyword);
}
}
You'd then have to call it as:
var filteredList = _contactRepository.SearchExact(keyword, cr => cr.Name).ToList();
Other than these two options you could always look into using reflection to access the Name property without any interface, but this has the downside that there's no compile-time check that makes sure the classes you're passing actually DO have a Name property and also has the side-effect that the LINQ will not be translated to SQL and the filtering will happen in .NET (Meaning the SQL server could get hit more than is needed).
You could also use a Dynamic LINQ query to achieve this SQL-side effect, but it has the same non type-safe issues listed above.