I'm using EF Core 3.1.1 (dotnet core 3.1.1). And I want to return a large number of Car entities. Unfortunately I get the following error message:
'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.Internal.InternalDbSet`...
I know that there is another answered question regarding the same error. But I'm not doing an explicit async operation.
[HttpGet]
[ProducesResponseType(200, Type = typeof(Car[]))]
public IActionResult Index()
{
return Ok(_carsDataModelContext.Cars.AsEnumerable());
}
The _carDataModelContext.Car is just a simple entity that maps 1-on-1 to a table in the database. public virtual DbSet<Car> Cars { get; set; }
Originally I return Ok(_carsDataModelContext.Cars.AsQueryable()) because we need to support OData. But to be sure it wasn't OData that is messing things up I tried to return AsEnumerable, and remove the "[EnableQuery]" attribute from the method. But that still ends in the same error.
The only way to fix this, is if I return Ok(_carsDataModelContext.Cars.ToList())
All Ef Core IQueryable<T> implementations (DbSet<T>, EntityQueryable<T>) also implement the standard IAsyncEnumerable<T> interface (when used from .NET Core 3), so AsEnumerable(), AsQueryable() and AsAsyncEnumerable() simply return the same instance cast to the corresponding interface.
You can easily verify that with the following snippet:
var queryable = _carsDataModelContext.Cars.AsQueryable();
var enumerable = queryable.AsEnumerable();
var asyncEnumerable = queryable.AsAsyncEnumerable();
Debug.Assert(queryable == enumerable && queryable == asyncEnumerable);
So even though you are not returning explicitly IAsyncEnumerable<T>, the underlying object implements it and can be queried for. Knowing that Asp.Net Core is naturally async framework, we can safely assume that it checks if the object implements the new standard IAsyncEnumerable<T>, and uses that behind the scenes instead of IEnumerable<T>.
Of course when you use ToList(), the returned List<T> class does not implement IAsyncEnumerable<T>, hence the only option is to use IEnumerable<T>.
This should explain the 3.1 behavior. Note that before 3.0 there was no standard IAsyncEnumerable<T> interface. EF Core was implementing and returning its own async interface, but the .Net Core infrastructure was unaware of it, thus was unable to use it on behalf of you.
The only way to force the previous behavior without using ToList() / ToArray() and similar is to hide the underlying source (hence the IAsyncEnumerable<T>).
For IEnumerable<T> it's quite easy. All you need is to create custom extension method which uses C# iterator, e.g:
public static partial class Extensions
{
public static IEnumerable<T> ToEnumerable<T>(this IEnumerable<T> source)
{
foreach (var item in source)
yield return item;
}
}
and then use
return Ok(_carsDataModelContext.Cars.ToEnumerable());
If you want to return IQueryable<T>, the things get harder. Creating custom IQueryable<T> wrapper is not enough, you have to create custom IQueryProvider wrapper to make sure composing over returned wrapped IQueryable<T> would continue returning wrappers until the final IEnumerator<T> (or IEnumerator) is requested, and the returned underlying async enumerable is hidden with the aforementioned method.
Here is a simplified implementation of the above:
public static partial class Extensions
{
public static IQueryable<T> ToQueryable<T>(this IQueryable<T> source)
=> new Queryable<T>(new QueryProvider(source.Provider), source.Expression);
class Queryable<T> : IQueryable<T>
{
internal Queryable(IQueryProvider provider, Expression expression)
{
Provider = provider;
Expression = expression;
}
public Type ElementType => typeof(T);
public Expression Expression { get; }
public IQueryProvider Provider { get; }
public IEnumerator<T> GetEnumerator() => Provider.Execute<IEnumerable<T>>(Expression)
.ToEnumerable().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
class QueryProvider : IQueryProvider
{
private readonly IQueryProvider source;
internal QueryProvider(IQueryProvider source) => this.source = source;
public IQueryable CreateQuery(Expression expression)
{
var query = source.CreateQuery(expression);
return (IQueryable)Activator.CreateInstance(
typeof(Queryable<>).MakeGenericType(query.ElementType),
this, query.Expression);
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
=> new Queryable<TElement>(this, expression);
public object Execute(Expression expression) => source.Execute(expression);
public TResult Execute<TResult>(Expression expression) => source.Execute<TResult>(expression);
}
}
The query provider implementation is not fully correct, because it assumes that only the custom Queryable<T> will call Execute methods for creating IEnumerable<T>, and external calls will be used only for immediate methods like Count, FirstOrDefault, Max etc., but it should work for this scenario.
Other drawback of this implementation is that all EF Core specific Queryable extensions won't work, which might be an issue/showstopper if OData $expand relies on methods like Include / ThenInclude. But fixing that requires more complex implementation digging into EF Core internals.
With that being said, the usage of course would be:
return Ok(_carsDataModelContext.Cars.ToQueryable());
Related
I have a data layer which abstracts away the underlying implementation (Entity Framework) by accepting & returning models that are defined elsewhere. I want to be able to pass in a Func<T, bool> predicate to one of the methods to allow additional clauses to be applied when querying the database.
Since the generic models know nothing of the underlying Entity Framework implementation, I need to convert my Func<T, bool> predicate to a predicate which can be applied against the Entity Framework context.
The generic models & Entity Framework models have exactly the same property names & I already have a class to perform the property value mappings between the two. Is there a way to convert the target type of the generic model delegate to that of the Entity Framework model?
An example of what I'm attempting to do:
Given this delegate: func = Func<Schema.Data.Order, bool>
I want to convert it to: dbFunc = Func<Models.Order, bool>
And apply it to the context: ctx.Orders.Where(dbDel)
I found this post, but I can't piece together how to perform the conversion. Been bumping my head against this the whole evening, so any help would be greatly appreciated!
UPDATE
The original question & requirements seem to have been a bit vague, so I'll elaborate on my implementation & my requirements. The following code samples have been amended to use Expression<Func<TIn, TOut>> & not Func<TIn, TOut>, based on the suggestions from hvd & Alexei.
I have an interface & a set of classes that represent my data layer. These act as a façade over the data source, allowing for different implementations in order to access the database. I want to be able to pass through additional filter criteria as a predicate & apply this to the underlying data source. But since the façade is separate from the underlying implementation, the predicate is defined using the façade model classes. Furthermore, the façade model class properties have the same naming as that of my implementation, so direct property assignments using reflection, for example, is possible.
My façade implementation:
namespace Schema.Data
{
public interface IDataStore
{
public IEnumerable<Order> GetOrders(string custCode, Expression<Func<Order, bool>> exp);
}
public class Order
{
public string CustomerCode { get; set; }
public string OrderNumber { get; set; }
}
}
I then implement the interface in a separate namespace, using Entity Framework to query my database:
namespace Data.EF
{
// Entity Framework model is in this same namespace
public class DataStore : Schema.Data.IDataStore
{
public IEnumerable<Schema.Data.Order> GetOrders(string custCode, Expression<Func<Schema.Data.Order, bool>> exp)
{
using (var ctx = new MyDatabaseEntities()) {
// TODO: Convert "exp" to Expression<Func<Data.EF.Order, bool>> to pass it in below
var orders = ctx.Orders.Where(e => e.CustomerCode == custCode).Where(dbExp ?? (n => true));
// handling the retrieved data & returning result goes here
}
}
}
}
If you just need to convert function taking one type to function taking another type use regular composition approach -
TDest MapSourceToDest(TSource v) {....}
Func<TSource, bool> func = ...;
Func<TDest, bool> dbFunc = x => func(MapSourceToDest(x));
Mapping can be done by hand, reflection, libraries or many other approaches - How to map properties of two different objects?.
Note that if you actually need to pass such method to EF you need Expression and not Func - in this case it actually possible to rewrite expression with destination type - How to change a type in an expression tree?
Wasn't really sure how to phrase the title.
What I am trying to achieve is a deep clone system for IEnumerable<T>s where T:ICloneable.
I have written the, as-yet untested, method below which I believe should work:
public static IEnumerable<T> DeepClone<T>(this IEnumerable<T> source) where T:ICloneable
{
return source.Select(s => (T) s.Clone());
}
However, this returns an IEnumerable<T> (as one would expect) and I am curious as to whether or not it is possible (without causing an unacceptable overhead) to return the base type of the IEnumerable<T> instead.
For example, running List<int>.DeepClone() would return a new, cloned List<int> and running int[].DeepClone() would return a new, cloned int[].
I know that I can quite easily just cast my IEnumerables after calling this method, but I'm hoping to be able to avoid this.
There is also the option of creating a whole load of overloads, one for each IEnumerable but if it's possible to I'd like to avoid this.
You will need to build explicit methods for the concrete types you want to support (List, arrays etc).
An example:
public static List<T> DeepClone<T>(this List<T> source) where T : ICloneable
{
return source.Select(s => (T)s.Clone()).ToList();
}
Alternatively, use an approach like:
public static IEnumerable<T> DeepClone<T>(this IEnumerable<T> source) where T : ICloneable
{
var result = source.Select(s => (T)s.Clone());
if (source is List<T>)
{
return result.ToList();
}
return result;
}
I have the following method in my generic class:
// This is the class declaration
public abstract class BaseService<TEntity, TKey> : IBaseService<TEntity, TKey> where TEntity : class, IEntity<TKey>
// The Method
public IQueryable<TEntity> GetActive()
{
if (typeof(IActivable).IsAssignableFrom(typeof(TEntity)))
{
return this.repository.Get().Cast<IActivable>()
.Where(q => q.Active)
.Cast<TEntity>();
}
else
{
return this.Get();
}
}
This is the interface:
public interface IActivable
{
bool Active { get; set; }
}
Basically, TEntity is an Entity (POCO) class, that can implement IActivable if they have Active property. I want to the method to return all records that have Active value true. However, I have this error:
Unable to cast the type 'WebTest.Models.Entities.Product' to type
'Data.IActivable'. LINQ to Entities only supports casting EDM
primitive or enumeration types.
I understand why this error occurs. But the articles on SO does not have any valid solution for my case. Is it achievable with Cast, or any other way? Note: I do not want to convert to IEnumerable, I want to keep IQueryable.
The EF expression parser will work without casting, however you won't be able to compile the C# code without the casting (C# will complain that it doesn't know that TEntity has an Active property). The solution is: cast for the c# compiler and not cast for the EF expression parser.
So if you are sure (you are checking it in the if, so you are) that the object implements IActivable, you can create the expression with the casting (for compiling) and then remove the castings in runtime (which are unnecessary) for EF. For your particular case:
public IQueryable<TEntity> GetActive()
{
if (typeof(IActivable).IsAssignableFrom(typeof(TEntity)))
{
Expression<Func<TEntity, bool>> getActive = x => ((IActivable)x).Active;
getActive = (Expression<Func<TEntity, bool>>)RemoveCastsVisitor.Visit(getActive);
return this.repository.Get().Where(getActive);
}
else
{
return this.Get();
}
}
The expression visitor is implemented like this:
internal class RemoveCastsVisitor : ExpressionVisitor
{
private static readonly ExpressionVisitor Default = new RemoveCastsVisitor();
private RemoveCastsVisitor()
{
}
public new static Expression Visit(Expression node)
{
return Default.Visit(node);
}
protected override Expression VisitUnary(UnaryExpression node)
{
if (node.NodeType == ExpressionType.Convert
&& node.Type.IsAssignableFrom(node.Operand.Type))
{
return base.Visit(node.Operand);
}
return base.VisitUnary(node);
}
}
It just checks if a casting is necessary: if the actual value already implements the type it's casting to, it'll just remove the conversion from the expression, and EF will pick it up correctly.
The trick is to cast the whole IQueryable<TEntity> to IQueryable<IActivable> instead of the first cast:
if (typeof(IActivable).IsAssignableFrom(typeof(TEntity)))
{
return ((IQueryable<IActivable>)(this.repository.Get()))
.Where(q => q.Active)
.Cast<TEntity>();
}
Currently I have an alternative is to use Extension method. However the downside is that my IBaseService cannot declare the GetActive method because the concrete classes do not actually implement it.
public static class BaseServiceExtension
{
public static IQueryable<TEntity> GetActive<TEntity, TKey>(this IBaseService<TEntity, TKey> service)
where TEntity : class, IEntity<TKey>, IActivable
{
return service.Get().Where(q => q.Active);
}
}
I would like to pass an IQueryable and an array of ids to a method which filters the IQueryable based on those ids.
As the ids can be either long's or int's it should be solved generically.
I came up with the following:
public static IEnumerable<T> GetModified<TId, T>(IQueryable<T> objects, TId[] ids) where T : class
{
return objects.Where(j => ids.Contains((TId)j.GetType().GetProperty("Id").GetValue(j)));
}
Unfortunately I'm getting the exception:
LINQ to Entities does not recognize the method 'System.Object GetValue(System.Object)' method, and this method cannot be translated into a store expression.
The exception is normal, as getting properties through reflection is something that clearly cannot be translated to SQL.
One thing I would try is to create a generic interface that exposes an Id property of a given type:
public interface HasId<T> {
T Id { get; set; }
}
Now you could declare your entity as implementing HasId<int>, for example, if the Id was of type int.
The next step is to modify your method like so:
public static IEnumerable<T> GetModified<TId, T>
(IQueryable<T> objects, TId[] ids) where T : class, HasId<TId>
{
return objects.Where(j => ids.Contains(j.Id));
}
Note the added generic restriction: where T : class, HasId<TId>. This enables you to write the simplified j.Id, which returns a TId value, instead of resorting to reflection.
Please note that I haven't run or tested this code; it's just an idea that I got when I saw your problem and I hope it helps.
Update:
Here's another possible solution that doesn't require that you declare interfaces or change your classes in any way:
public static IEnumerable<T> GetModified<TId, T>
(IQueryable<T> objects, TId[] ids, Expression<Func<T, TId>> idSelector)
where T : class
{
return objects.Where(j => ids.Contains(idSelector(j)));
}
What I've done here is add the Expression<Func<T, TId>> idSelector parameter, an expression that can return the Id of a given instance of T.
You would call the method like that:
var modified = GetModified(dbObjects, yourIdArray, entity => entity.Id);
(only the third parameter being new; keep the others as you have them now).
Again, I haven't tested if this works or even compiles, as I don't have a computer with VS here :(.
Entity Framework doesn't support some of the .NET methods such as GetValue() since it does not translate to SQL (which is the code actually executed to the IQueryable. Try calling ToList to get the CLR object before doing reflection:
public static IEnumerable<T> GetModified<TId, T>(IQueryable<T> objects, TId[] ids) where T : class
{
return objects.ToList().Where(j => ids.Contains((TId)j.GetType().GetProperty("Id").GetValue(j)));
}
Note: Please re-tag and/or re-name appropriately
I have a class, FooEnumerator, that wraps a Foo and implements IEnumerable<FooEnumerator>. The Foos represent a tree-like data structure, the FooEnumerators that are enumerated are the child nodes of the current node.
Foo is a vendor supplied data object. FooEnumerator implements a bunch of custom filtering code.
class FooEnumerator : IEnumerable<FooEnumerator>
{
public Foo WrappedNode { get; private set; }
public string Name { get { return WrappedNode.Name; } }
public int Id { get{ return WrappedNode.Id; } }
public DateTime Created { get{ return WrappedNode.Created; } }
public FooEnumerator(Foo wrappedNode)
{
WrappedNode = wrappedNode;
}
public IEnumerator<FooEnumerator> GetEnumerator()
{
foreach (Foo child in this.GetChildren())
if(FilteringLogicInHere(child))
yield return new FooEnumerator(child);
}
...
}
I want to be able to sort each level of the tree with a given (arbitrary) expression, defined when the top level FooEnumerator is created, and have this expression passed down to each newly enumerated item to use.
I'd like to define the sort expression using lambda's, in the same way you would with the OrderBy function. In fact, it is my intention to pass the lambda to OrderBy.
The signiture for OrderBy is
OrderBy<TSource, TKey>(Func<TSource, TKey> keySelector)
where TKey is the return type of the given Func, but is a Type Parameter in the method signature and is figured out at compile time.
Example usage
var x = GetStartingNode();
var sort = n => n.DateTime;
var enu = new FooEnumerator(x, sort);
var sort2 = n => n.Name;
var enu2 = new FooEnumerator(x, sort2);
The sort expression would then be stored in a class variable and FooEnumerator would work like:
// pseudo-implementation
private Expression<Func<Foo, TKey>> _sortBy;
public FooEnumerator(Foo wrappedNode, Expression<Func<Foo, TKey>> sortBy)
{
WrappedNode = wrappedNode;
_sortBy = sortBy;
}
public IEnumerator<FooEnumerator> GetEnumerator()
{
foreach (Foo child in this.GetChildren().OrderBy(_sortBy))
if(FilteringLogicInHere(child))
yield return new FooEnumerator(child);
}
How can I specify the type of TKey (implicitly or explicitly) in this use case?
I don't want to hard code it as I want to be able to sort on any and all properties of the underlying Foo.
Well, you can't create a member delegate variable of type Expression<Func<Foo,TKey>> since TKey is never specified. However, you could create a member of type Expression<Func<Foo,IComparable>> which may suffice for your purposes. You could need to change your FooEnumerator constructor to accept this signature as well, of course.
EDIT: Others have suggested parameterizing your FooEnumerator so that it accepts a TKey. You can certainly do this, but you should be aware of the issues that emerge:
By parameterizing the enumerator you are then kicking the bucket down the road. Any code that wants to store a FooEnumerator<T> has to have a-priori knowledge of the type T. You could, however, implement a non-generic interface IFooEnumerator to deal with that.
Parameterizing an enumerator creates issues if you want to support ordering on multiple fields in the future. C# doesn't support generics with a variable number of type parameters, which limits the creation of generics that require multiple arbitrary types. This issue is harder to deal with, since it's awkward to start creating FooEnumerator<T>, FooEnumerator<T1,T2>, FooEnumerator<T1,T2,T3...>, and so on.
You can also parameterize your Enumerator:
class FooEnumerator<TKey> {
// ... All your 'pseudo' code would work here
}
I recommend programming against the interface using IComparable however.