Related
As commonly known in EF-Core there is no Lazy loading. So that kind of means I'm forced to do my queries with some afterthought. So since I have to think, then i might as well try to do it properly.
I have a Fairly standard update query, but I thought hey, I don't always have to include the HeaderImage and PromoImage FK-objects. There should be a way to make that happen. But I can just not find a way to perform a Include at a later point. In-fact I would like to maybe include right before I actually do work on the object. That way i might be able to automate some of the eagerness.
ArticleContent ac = _ctx.ArticleContents
.Include(a=> a.Metadata)
.Include(a=> a.HeaderImage)
.Include(a=> a.PromoImage)
.Single(a => a.Id == model.BaseID);
ac.Title = model.Title;
ac.Ingress = model.Ingress;
ac.Body = model.Body;
ac.Footer = model.Footer;
if (model.HeaderImage != null)
{
ac.HeaderImage.FileURL = await StoreImage(model.HeaderImage, $"Header_{model.Title.Replace(" ", "_")}_{rand.Next()}");
}
if (model.PromoImage != null)
{
ac.PromoImage.FileURL = await StoreImage(model.PromoImage, $"Promo_{model.Title.Replace(" ", "_")}_{rand.Next()}");
}
ac.Metadata.EditedById = uId;
ac.Metadata.LastChangedTimestamp = DateTime.Now;
await _ctx.SaveChangesAsync();
EXTRA
To be clear, this is EF7 (Core), and im after a solution that allows me to add includes on demand, hopefully after the initial _ctx.ArticleContents.Include(a=> a.Metadata).Single(a => a.Id == model.BaseID).
I'm using something similar to Alexander Derck's solution. (Regarding the exception mentioned in the comments: ctx.ArticleContents.AsQueryable() should also work.)
For a couple of CRUD MVC sites I'm using a BaseAdminController. In the derived concrete Controllers I can add Includes dynamically. From the BaseAdminController:
// TModel: e.g. ArticleContent
private List<Expression<Func<TModel, object>>> includeIndexExpressionList = new List<Expression<Func<TModel, object>>>();
protected void AddIncludes(Expression<Func<TModel, object>> includeExpression)
{
includeIndexExpressionList.Add(includeExpression);
}
Later I saw that I need more flexibility, so I added a queryable. E.g. for ThenInclude().
private Func<IQueryable<TModel>, IQueryable<TModel>> IndexAdditionalQuery { get; set; }
protected void SetAdditionalQuery(Func<IQueryable<TModel>, IQueryable<TModel>> queryable)
{
IndexAdditionalQuery = queryable;
}
Here the Index action:
public virtual async Task<IActionResult> Index()
{
// dynamic include:
// dbset is for instance ctx.ArticleContents
var queryable = includeIndexExpressionList
.Aggregate(dbSet.AsQueryable(), (current, include) => current.Include(include));
if(IndexAdditionalQuery != null) queryable = IndexAdditionalQuery(queryable);
var list = await queryable.Take(100).AsNoTracking().ToListAsync();
var viewModelList = list.Map<IList<TModel>, IList<TViewModel>>();
return View(viewModelList);
}
In the concrete Controller I use:
AddIncludes(e => e.EventCategory);
SetAdditionalQuery(q => q
.Include(e => e.Event2Locations)
.ThenInclude(j => j.Location));
I would create a method to fetch the data which takes the properties to include as expressions:
static ArticleContent GetArticleContent(int ID,
params Expression<Func<ArticleContent, object>>[] includes)
{
using(var ctx = new MyContext())
{
var acQuery = _ctx.ArticleContents.Include(a=> a.Metadata);
foreach(var include in includes)
acQuery = acQuery.Include(include);
return acQuery.Single(a => a.Id == model.BaseID);
}
}
And then in main method:
var ac = GetArticleContent(3, a => a.HeaderImage, a => a.PromoImage);
I am trying to create a generic repository to access my database. In EF6 I was able to do that in order to get a specific entity:
protected IDbSet<T> dbset;
public T Get(object id)
{
return this.dbset.Find(id);
}
DbSet in EF7 is missing a Find method. Is there a way to implement the above piece of code?
Here's a very crude, incomplete, and untested implementation of .Find() as an extension method. If nothing else, it should get you pointed in the right direction.
The real implementation is tracked by #797.
static TEntity Find<TEntity>(this DbSet<TEntity> set, params object[] keyValues)
where TEntity : class
{
var context = ((IAccessor<IServiceProvider>)set).Service.GetService<DbContext>();
var entityType = context.Model.GetEntityType(typeof(TEntity));
var key = entityType.GetPrimaryKey();
var entries = context.ChangeTracker.Entries<TEntity>();
var i = 0;
foreach (var property in key.Properties)
{
var keyValue = keyValues[i];
entries = entries.Where(e => e.Property(property.Name).CurrentValue == keyValue);
i++;
}
var entry = entries.FirstOrDefault();
if (entry != null)
{
// Return the local object if it exists.
return entry.Entity;
}
// TODO: Build the real LINQ Expression
// set.Where(x => x.Id == keyValues[0]);
var parameter = Expression.Parameter(typeof(TEntity), "x");
var query = set.Where((Expression<Func<TEntity, bool>>)
Expression.Lambda(
Expression.Equal(
Expression.Property(parameter, "Id"),
Expression.Constant(keyValues[0])),
parameter));
// Look in the database
return query.FirstOrDefault();
}
In case you are using EF 7.0.0-rc1-final, below you find a small update for the code presented by #bricelam in the previous answer. By the way, thank you very much #bricelam - your code was extremely useful for me.
Here are my dependencies under "project.config":
"dependencies": {
"EntityFramework.Commands": "7.0.0-rc1-final",
"EntityFramework.MicrosoftSqlServer": "7.0.0-rc1-final",
"Microsoft.Framework.Configuration.Json": "1.0.0-beta8",
"Microsoft.Framework.ConfigurationModel": "1.0.0-beta4",
"Microsoft.Framework.ConfigurationModel.Json": "1.0.0-beta4",
"Microsoft.Framework.DependencyInjection": "1.0.0-beta8"
}
And below is the extension method for DbSet.Find(TEntity):
using Microsoft.Data.Entity;
using Microsoft.Data.Entity.Infrastructure;
using System;
using System.Linq;
using System.Linq.Expressions;
namespace Microsoft.Data.Entity.Extensions
{
public static class Extensions
{
public static TEntity Find<TEntity>(this DbSet<TEntity> set, params object[] keyValues) where TEntity : class
{
var context = ((IInfrastructure<IServiceProvider>)set).GetService<DbContext>();
var entityType = context.Model.FindEntityType(typeof(TEntity));
var key = entityType.FindPrimaryKey();
var entries = context.ChangeTracker.Entries<TEntity>();
var i = 0;
foreach (var property in key.Properties)
{
entries = entries.Where(e => e.Property(property.Name).CurrentValue == keyValues[i]);
i++;
}
var entry = entries.FirstOrDefault();
if (entry != null)
{
// Return the local object if it exists.
return entry.Entity;
}
// TODO: Build the real LINQ Expression
// set.Where(x => x.Id == keyValues[0]);
var parameter = Expression.Parameter(typeof(TEntity), "x");
var query = set.Where((Expression<Func<TEntity, bool>>)
Expression.Lambda(
Expression.Equal(
Expression.Property(parameter, "Id"),
Expression.Constant(keyValues[0])),
parameter));
// Look in the database
return query.FirstOrDefault();
}
}
}
Can't comment because of reputation, but if you use RC2 (or later?) you should use
var context = set.GetService<ICurrentDbContext>().Context;
instead of
var context = set.GetService<DbContext>();
I've taken some of the previously provided answers and tweaked them to fix a couple of problems:
Implicitly captured closure
Key shouldn't be hard coded to "Id"
public static TEntity Find<TEntity>(this DbSet<TEntity> set, params object[] keyValues) where TEntity : class
{
var context = set.GetService<DbContext>();
var entityType = context.Model.FindEntityType(typeof(TEntity));
var key = entityType.FindPrimaryKey();
var entries = context.ChangeTracker.Entries<TEntity>();
var i = 0;
foreach (var property in key.Properties)
{
var i1 = i;
entries = entries.Where(e => e.Property(property.Name).CurrentValue == keyValues[i1]);
i++;
}
var entry = entries.FirstOrDefault();
if (entry != null)
{
// Return the local object if it exists.
return entry.Entity;
}
var parameter = Expression.Parameter(typeof(TEntity), "x");
var query = set.AsQueryable();
i = 0;
foreach (var property in key.Properties)
{
var i1 = i;
query = query.Where((Expression<Func<TEntity, bool>>)
Expression.Lambda(
Expression.Equal(
Expression.Property(parameter, property.Name),
Expression.Constant(keyValues[i1])),
parameter));
i++;
}
// Look in the database
return query.FirstOrDefault();
}
Find finally arrives into Entity Framework core.
So...the above find methods worked great, but if you don't have a column named "Id" in your model, the whole thing is going to fail on the following line. I'm not sure why the OP would have put a hardcoded value into this spot
Expression.Property(parameter, "Id"),
Here's a revision that will fix it for those that name our Id columns appropriately. :)
var keyCompare = key.Properties[0].Name;
// TODO: Build the real LINQ Expression
// set.Where(x => x.Id == keyValues[0]);
var parameter = Expression.Parameter(typeof(TEntity), "x");
var query = set.Where((Expression<Func<TEntity, bool>>)
Expression.Lambda(
Expression.Equal(
Expression.Property(parameter, keyCompare),
//Expression.Property(parameter, "Id"),
Expression.Constant(keyValues[0])),
parameter));
// Look in the database
return query.FirstOrDefault();
}
This STILL very well could fail if you have more than one Key setup on the entity object and the key you're looking up by isn't the first, but it should be quite a bit btter this way.
Not enough reputation to comment, but there is a bug in #Roger-Santana answer when using it in a console app/seperate assembly:
var i = 0;
foreach (var property in key.Properties)
{
entries = entries.Where(e => e.Property(property.Name).CurrentValue == keyValues[i]);
i++;
}
var entry = entries.FirstOrDefault();
The value of 'i' is captured in the foreach so that when entries.FirstOrDefault() is called, keyValues[i] has the value of (at least) keyValues[i++], which in my case crashed with an out of index error.
A fix would be to copy the value of 'i' through the loop:
var i = 0;
foreach (var property in key.Properties)
{
var idx =i;
entries = entries.Where(e => e.Property(property.Name).CurrentValue == keyValues[idx]);
i++;
}
var entry = entries.FirstOrDefault();
I use linq; instead of Find method you can use:
var record = dbSet.SingleOrDefault(m => m.Id == id)
Let me contribute a revision that includes building the expression.
I'll confess I didn't actually test this ;-)
public static TEntity Find<TEntity>(this DbSet<TEntity> dbSet, params object[] keyValues) where TEntity : class
{
// Find DbContext, entity type, and primary key.
var context = ((IInfrastructure<IServiceProvider>)dbSet).GetService<DbContext>();
var entityType = context.Model.FindEntityType(typeof(TEntity));
var key = entityType.FindPrimaryKey();
// Build the lambda expression for the query: (TEntity entity) => AND( entity.keyProperty[i] == keyValues[i])
var entityParameter = Expression.Parameter(typeof(TEntity), "entity");
Expression whereClause = Expression.Constant(true, typeof(bool));
uint i = 0;
foreach (var keyProperty in key.Properties) {
var keyMatch = Expression.Equal(
Expression.Property(entityParameter, keyProperty.Name),
Expression.Constant(keyValues[i++])
);
whereClause = Expression.And(whereClause, keyMatch);
}
var lambdaExpression = (Expression<Func<TEntity,bool>>)Expression.Lambda(whereClause, entityParameter);
// Execute against the in-memory entities, which we get from ChangeTracker (but not filtering the state of the entities).
var entries = context.ChangeTracker.Entries<TEntity>().Select((EntityEntry e) => (TEntity)e.Entity);
TEntity entity = entries.AsQueryable().Where(lambdaExpression).First(); // First is what triggers the query execution.
// If found in memory then we're done.
if (entity != null) { return entity; }
// Otherwise execute the query against the database.
return dbSet.Where(lambdaExpression).First();
}
here is what I use.
Not a find method, but works like a charm
var professionalf = from m in _context.Professionals select m;
professionalf = professionalf.Where(s => s.ProfessionalId == id);
Professional professional = professionalf.First();
An edit was proposed to change ".First()" to ".FirstOrDefault()" in the very last line of my earlier post. The edit was voted down, but I agree with it. I would expect the function to return null if the key was not found. I would not want it to throw an exception. In most cases I would want to know if the key existed in the set, and handling an exception is a very slow way of figuring that out.
given a list of ids, I can query all relevant rows by:
context.Table.Where(q => listOfIds.Contains(q.Id));
But how do you achieve the same functionality when the Table has a composite key?
This is a nasty problem for which I don't know any elegant solution.
Suppose you have these key combinations, and you only want to select the marked ones (*).
Id1 Id2
--- ---
1 2 *
1 3
1 6
2 2 *
2 3 *
... (many more)
How to do this is a way that Entity Framework is happy? Let's look at some possible solutions and see if they're any good.
Solution 1: Join (or Contains) with pairs
The best solution would be to create a list of the pairs you want, for instance Tuples, (List<Tuple<int,int>>) and join the database data with this list:
from entity in db.Table // db is a DbContext
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
In LINQ to objects this would be perfect, but, too bad, EF will throw an exception like
Unable to create a constant value of type 'System.Tuple`2 (...) Only primitive types or enumeration types are supported in this context.
which is a rather clumsy way to tell you that it can't translate this statement into SQL, because Tuples is not a list of primitive values (like int or string). For the same reason a similar statement using Contains (or any other LINQ statement) would fail.
Solution 2: In-memory
Of course we could turn the problem into simple LINQ to objects like so:
from entity in db.Table.AsEnumerable() // fetch db.Table into memory first
join pair Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
Needless to say that this is not a good solution. db.Table could contain millions of records.
Solution 3: Two Contains statements (incorrect)
So let's offer EF two lists of primitive values, [1,2] for Id1 and [2,3] for Id2. We don't want to use join, so let's use Contains:
from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity
But now the results also contains entity {1,3}! Well, of course, this entity perfectly matches the two predicates. But let's keep in mind that we're getting closer. In stead of pulling millions of entities into memory, we now only get four of them.
Solution 4: One Contains with computed values
Solution 3 failed because the two separate Contains statements don't only filter the combinations of their values. What if we create a list of combinations first and try to match these combinations? We know from solution 1 that this list should contain primitive values. For instance:
var computed = ids1.Zip(ids2, (i1,i2) => i1 * i2); // [2,6]
and the LINQ statement:
from entity in db.Table
where computed.Contains(entity.Id1 * entity.Id2)
select entity
There are some problems with this approach. First, you'll see that this also returns entity {1,6}. The combination function (a*b) does not produce values that uniquely identify a pair in the database. Now we could create a list of strings like ["Id1=1,Id2=2","Id1=2,Id2=3]" and do
from entity in db.Table
where computed.Contains("Id1=" + entity.Id1 + "," + "Id2=" + entity.Id2)
select entity
(This would work in EF6, not in earlier versions).
This is getting pretty messy. But a more important problem is that this solution is not sargable, which means: it bypasses any database indexes on Id1 and Id2 that could have been used otherwise. This will perform very very poorly.
Solution 5: Best of 2 and 3
So the most viable solution I can think of is a combination of Contains and a join in memory: First do the contains statement as in solution 3. Remember, it got us very close to what we wanted. Then refine the query result by joining the result as an in-memory list:
var rawSelection = from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity;
var refined = from entity in rawSelection.AsEnumerable()
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity;
It's not elegant, messy all the same maybe, but so far it's the only scalable1 solution to this problem I found, and applied in my own code.
Solution 6: Build a query with OR clauses
Using a Predicate builder like Linqkit or alternatives, you can build a query that contains an OR clause for each element in the list of combinations. This could be a viable option for really short lists. With a couple of hundreds of elements, the query will start performing very poorly. So I don't consider this a good solution unless you can be 100% sure that there will always be a small number of elements. One elaboration of this option can be found here.
Solution 7: Unions
There's also a solution using UNIONs that I posted later here.
1As far as the Contains statement is scalable: Scalable Contains method for LINQ against a SQL backend
Solution for Entity Framework Core with SQL Server
🎉 NEW! QueryableValues EF6 Edition has arrived!
The following solution makes use of QueryableValues. This is a library that I wrote to primarily solve the problem of query plan cache pollution in SQL Server caused by queries that compose local values using the Contains LINQ method. It also allows you to compose values of complex types in your queries in a performant way, which will achieve what's being asked in this question.
First you will need to install and set up the library, after doing that you can use any of the following patterns that will allow you to query your entities using a composite key:
// Required to make the AsQueryableValues method available on the DbContext.
using BlazarTech.QueryableValues;
// Local data that will be used to query by the composite key
// of the fictitious OrderProduct table.
var values = new[]
{
new { OrderId = 1, ProductId = 10 },
new { OrderId = 2, ProductId = 20 },
new { OrderId = 3, ProductId = 30 }
};
// Optional helper variable (needed by the second example due to CS0854)
var queryableValues = dbContext.AsQueryableValues(values);
// Example 1 - Using a Join (preferred).
var example1Results = dbContext
.OrderProduct
.Join(
queryableValues,
e => new { e.OrderId, e.ProductId },
v => new { v.OrderId, v.ProductId },
(e, v) => e
)
.ToList();
// Example 2 - Using Any (similar behavior as Contains).
var example2Results = dbContext
.OrderProduct
.Where(e => queryableValues
.Where(v =>
v.OrderId == e.OrderId &&
v.ProductId == e.ProductId
)
.Any()
)
.ToList();
Useful Links
Nuget Package
GitHub Repository
Benchmarks
QueryableValues is distributed under the MIT license.
You can use Union for each composite primary key:
var compositeKeys = new List<CK>
{
new CK { id1 = 1, id2 = 2 },
new CK { id1 = 1, id2 = 3 },
new CK { id1 = 2, id2 = 4 }
};
IQuerable<CK> query = null;
foreach(var ck in compositeKeys)
{
var temp = context.Table.Where(x => x.id1 == ck.id1 && x.id2 == ck.id2);
query = query == null ? temp : query.Union(temp);
}
var result = query.ToList();
You can create a collection of strings with both keys like this (I am assuming that your keys are int type):
var id1id2Strings = listOfIds.Select(p => p.Id1+ "-" + p.Id2);
Then you can just use "Contains" on your db:
using (dbEntities context = new dbEntities())
{
var rec = await context.Table1.Where(entity => id1id2Strings .Contains(entity.Id1+ "-" + entity.Id2));
return rec.ToList();
}
You need a set of objects representing the keys you want to query.
class Key
{
int Id1 {get;set;}
int Id2 {get;set;}
If you have two lists and you simply check that each value appears in their respective list then you are getting the cartesian product of the lists - which is likely not what you want. Instead you need to query the specific combinations required
List<Key> keys = // get keys;
context.Table.Where(q => keys.Any(k => k.Id1 == q.Id1 && k.Id2 == q.Id2));
I'm not completely sure that this is valid use of Entity Framework; you may have issues with sending the Key type to the database. If that happens then you can be creative:
var composites = keys.Select(k => p1 * k.Id1 + p2 * k.Id2).ToList();
context.Table.Where(q => composites.Contains(p1 * q.Id1 + p2 * q.Id2));
You can create an isomorphic function (prime numbers are good for this), something like a hashcode, which you can use to compare the pair of values. As long as the multiplicative factors are co-prime this pattern will be isomorphic (one-to-one) - i.e. the result of p1*Id1 + p2*Id2 will uniquely identify the values of Id1 and Id2 as long as the prime numbers are correctly chosen.
But then you end up in a situation where you're implementing complex concepts and someone is going to have to support this. Probably better to write a stored procedure which takes the valid key objects.
Ran into this issue as well and needed a solution that both did not perform a table scan and also provided exact matches.
This can be achieved by combining Solution 3 and Solution 4 from Gert Arnold's Answer
var firstIds = results.Select(r => r.FirstId);
var secondIds = results.Select(r => r.SecondId);
var compositeIds = results.Select(r => $"{r.FirstId}:{r.SecondId}");
var query = from e in dbContext.Table
//first check the indexes to avoid a table scan
where firstIds.Contains(e.FirstId) && secondIds.Contains(e.SecondId))
//then compare the compositeId for an exact match
//ToString() must be called unless using EF Core 5+
where compositeIds.Contains(e.FirstId.ToString() + ":" + e.SecondId.ToString()))
select e;
var entities = await query.ToListAsync();
For EF Core I use a slightly modified version of the bucketized IN method by EricEJ to map composite keys as tuples. It performs pretty well for small sets of data.
Sample usage
List<(int Id, int Id2)> listOfIds = ...
context.Table.In(listOfIds, q => q.Id, q => q.Id2);
Implementation
public static IQueryable<TQuery> In<TKey1, TKey2, TQuery>(
this IQueryable<TQuery> queryable,
IEnumerable<(TKey1, TKey2)> values,
Expression<Func<TQuery, TKey1>> key1Selector,
Expression<Func<TQuery, TKey2>> key2Selector)
{
if (values is null)
{
throw new ArgumentNullException(nameof(values));
}
if (key1Selector is null)
{
throw new ArgumentNullException(nameof(key1Selector));
}
if (key2Selector is null)
{
throw new ArgumentNullException(nameof(key2Selector));
}
if (!values.Any())
{
return queryable.Take(0);
}
var distinctValues = Bucketize(values);
if (distinctValues.Length > 1024)
{
throw new ArgumentException("Too many parameters for SQL Server, reduce the number of parameters", nameof(values));
}
var predicates = distinctValues
.Select(v =>
{
// Create an expression that captures the variable so EF can turn this into a parameterized SQL query
Expression<Func<TKey1>> value1AsExpression = () => v.Item1;
Expression<Func<TKey2>> value2AsExpression = () => v.Item2;
var firstEqual = Expression.Equal(key1Selector.Body, value1AsExpression.Body);
var visitor = new ReplaceParameterVisitor(key2Selector.Parameters[0], key1Selector.Parameters[0]);
var secondEqual = Expression.Equal(visitor.Visit(key2Selector.Body), value2AsExpression.Body);
return Expression.AndAlso(firstEqual, secondEqual);
})
.ToList();
while (predicates.Count > 1)
{
predicates = PairWise(predicates).Select(p => Expression.OrElse(p.Item1, p.Item2)).ToList();
}
var body = predicates.Single();
var clause = Expression.Lambda<Func<TQuery, bool>>(body, key1Selector.Parameters[0]);
return queryable.Where(clause);
}
class ReplaceParameterVisitor : ExpressionVisitor
{
private ParameterExpression _oldParameter;
private ParameterExpression _newParameter;
public ReplaceParameterVisitor(ParameterExpression oldParameter, ParameterExpression newParameter)
{
_oldParameter = oldParameter;
_newParameter = newParameter;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (ReferenceEquals(node, _oldParameter))
return _newParameter;
return base.VisitParameter(node);
}
}
/// <summary>
/// Break a list of items tuples of pairs.
/// </summary>
private static IEnumerable<(T, T)> PairWise<T>(this IEnumerable<T> source)
{
var sourceEnumerator = source.GetEnumerator();
while (sourceEnumerator.MoveNext())
{
var a = sourceEnumerator.Current;
sourceEnumerator.MoveNext();
var b = sourceEnumerator.Current;
yield return (a, b);
}
}
private static TKey[] Bucketize<TKey>(IEnumerable<TKey> values)
{
var distinctValueList = values.Distinct().ToList();
// Calculate bucket size as 1,2,4,8,16,32,64,...
var bucket = 1;
while (distinctValueList.Count > bucket)
{
bucket *= 2;
}
// Fill all slots.
var lastValue = distinctValueList.Last();
for (var index = distinctValueList.Count; index < bucket; index++)
{
distinctValueList.Add(lastValue);
}
var distinctValues = distinctValueList.ToArray();
return distinctValues;
}
In the absence of a general solution, I think there are two things to consider:
Avoid multi-column primary keys (will make unit testing easier too).
But if you have to, chances are that one of them will reduce the
query result size to O(n) where n is the size of the ideal query
result. From here, its Solution 5 from Gerd Arnold above.
For example, the problem leading me to this question was querying order lines, where the key is order id + order line number + order type, and the source had the order type being implicit. That is, the order type was a constant, order ID would reduce the query set to order lines of relevant orders, and there would usually be 5 or less of these per order.
To rephrase: If you have a composite key, changes are that one of them have very few duplicates. Apply Solution 5 from above with that.
I tried this solution and it worked with me and the output query was perfect without any parameters
using LinqKit; // nuget
var customField_Ids = customFields?.Select(t => new CustomFieldKey { Id = t.Id, TicketId = t.TicketId }).ToList();
var uniqueIds1 = customField_Ids.Select(cf => cf.Id).Distinct().ToList();
var uniqueIds2 = customField_Ids.Select(cf => cf.TicketId).Distinct().ToList();
var predicate = PredicateBuilder.New<CustomFieldKey>(false); //LinqKit
var lambdas = new List<Expression<Func<CustomFieldKey, bool>>>();
foreach (var cfKey in customField_Ids)
{
var id = uniqueIds1.Where(uid => uid == cfKey.Id).Take(1).ToList();
var ticketId = uniqueIds2.Where(uid => uid == cfKey.TicketId).Take(1).ToList();
lambdas.Add(t => id.Contains(t.Id) && ticketId.Contains(t.TicketId));
}
predicate = AggregateExtensions.AggregateBalanced(lambdas.ToArray(), (expr1, expr2) =>
{
var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
return Expression.Lambda<Func<CustomFieldKey, bool>>
(Expression.OrElse(expr1.Body, invokedExpr), expr1.Parameters);
});
var modifiedCustomField_Ids = repository.GetTable<CustomFieldLocal>()
.Select(cf => new CustomFieldKey() { Id = cf.Id, TicketId = cf.TicketId }).Where(predicate).ToArray();
I ended up writing a helper for this problem that relies on System.Linq.Dynamic.Core;
Its a lot of code and don't have time to refactor at the moment but input / suggestions appreciated.
public static IQueryable<TEntity> WhereIsOneOf<TEntity, TSource>(this IQueryable<TEntity> dbSet,
IEnumerable<TSource> source,
Expression<Func<TEntity, TSource,bool>> predicate) where TEntity : class
{
var (where, pDict) = GetEntityPredicate(predicate, source);
return dbSet.Where(where, pDict);
(string WhereStr, IDictionary<string, object> paramDict) GetEntityPredicate(Expression<Func<TEntity, TSource, bool>> func, IEnumerable<TSource> source)
{
var firstP = func.Parameters[0];
var binaryExpressions = RecurseBinaryExpressions((BinaryExpression)func.Body);
var i = 0;
var paramDict = new Dictionary<string, object>();
var res = new List<string>();
foreach (var sourceItem in source)
{
var innerRes = new List<string>();
foreach (var bExp in binaryExpressions)
{
var emp = ToEMemberPredicate(firstP, bExp);
var val = emp.GetKeyValue(sourceItem);
var pName = $"#{i++}";
paramDict.Add(pName, val);
var str = $"{emp.EntityMemberName} {emp.SQLOperator} {pName}";
innerRes.Add(str);
}
res.Add( "(" + string.Join(" and ", innerRes) + ")");
}
var sRes = string.Join(" || ", res);
return (sRes, paramDict);
}
EMemberPredicate ToEMemberPredicate(ParameterExpression firstP, BinaryExpression bExp)
{
var lMember = (MemberExpression)bExp.Left;
var rMember = (MemberExpression)bExp.Right;
var entityMember = lMember.Expression == firstP ? lMember : rMember;
var keyMember = entityMember == lMember ? rMember : lMember;
return new EMemberPredicate(entityMember, keyMember, bExp.NodeType);
}
List<BinaryExpression> RecurseBinaryExpressions(BinaryExpression e, List<BinaryExpression> runningList = null)
{
if (runningList == null) runningList = new List<BinaryExpression>();
if (e.Left is BinaryExpression lbe)
{
var additions = RecurseBinaryExpressions(lbe);
runningList.AddRange(additions);
}
if (e.Right is BinaryExpression rbe)
{
var additions = RecurseBinaryExpressions(rbe);
runningList.AddRange(additions);
}
if (e.Left is MemberExpression && e.Right is MemberExpression)
{
runningList.Add(e);
}
return runningList;
}
}
Helper class:
public class EMemberPredicate
{
public readonly MemberExpression EntityMember;
public readonly MemberExpression KeyMember;
public readonly PropertyInfo KeyMemberPropInfo;
public readonly string EntityMemberName;
public readonly string SQLOperator;
public EMemberPredicate(MemberExpression entityMember, MemberExpression keyMember, ExpressionType eType)
{
EntityMember = entityMember;
KeyMember = keyMember;
KeyMemberPropInfo = (PropertyInfo)keyMember.Member;
EntityMemberName = entityMember.Member.Name;
SQLOperator = BinaryExpressionToMSSQLOperator(eType);
}
public object GetKeyValue(object o)
{
return KeyMemberPropInfo.GetValue(o, null);
}
private string BinaryExpressionToMSSQLOperator(ExpressionType eType)
{
switch (eType)
{
case ExpressionType.Equal:
return "==";
case ExpressionType.GreaterThan:
return ">";
case ExpressionType.GreaterThanOrEqual:
return ">=";
case ExpressionType.LessThan:
return "<";
case ExpressionType.LessThanOrEqual:
return "<=";
case ExpressionType.NotEqual:
return "<>";
default:
throw new ArgumentException($"{eType} is not a handled Expression Type.");
}
}
}
Use Like so:
// This can be a Tuple or whatever.. If Tuple, then y below would be .Item1, etc.
// This data structure is up to you but is what I use.
[FromBody] List<CustomerAddressPk> cKeys
var res = await dbCtx.CustomerAddress
.WhereIsOneOf(cKeys, (x, y) => y.CustomerId == x.CustomerId
&& x.AddressId == y.AddressId)
.ToListAsync();
Hope this helps others.
in Case of composite key you can use another idlist and add a condition for that in your code
context.Table.Where(q => listOfIds.Contains(q.Id) && listOfIds2.Contains(q.Id2));
or you can use one another trick create a list of your keys by adding them
listofid.add(id+id1+......)
context.Table.Where(q => listOfIds.Contains(q.Id+q.id1+.......));
I tried this on EF Core 5.0.3 with the Postgres provider.
context.Table
.Select(entity => new
{
Entity = entity,
CompositeKey = entity.Id1 + entity.Id2,
})
.Where(x => compositeKeys.Contains(x.CompositeKey))
.Select(x => x.Entity);
This produced SQL like:
SELECT *
FROM table AS t
WHERE t.Id1 + t.Id2 IN (#__compositeKeys_0)),
Caveats
this should only be used where the combination of Id1 and Id2 will always produce a unique result (e.g., they're both UUIDs)
this cannot use indexes, though you could save the composite key to the db with an index
I am having trouble with the Select method in my repository method when I try to use certain return types.
The repository method where I am having the issue is:
public IEnumerable<T> List(Expression<Func<T, bool>> filter = null,
string include = "",
int Taked = 0, Expression<Func<T, T>> selector = null)
{
IQueryable<T> query = dbSet;
if (filter != null)
query = query.Where(filter);
#region Stringleri İnclude Eder
foreach (var includeProperty in
include.Split(new char[] {','},
StringSplitOptions.RemoveEmptyEntries))
{
query = query.Include(includeProperty);
}
#endregion
if (selector != null)
query = query.Select(selector);
if (Taked != 0)
return query.Take(Taked).ToList();
return query.ToList();
}
internal DbContext context;
internal DbSet<T> dbSet;
I want to return my entity class using the method above, but I only want certain properties to be populated. I have tried the following approach:
AdminWork workunit = new AdminWork();
IEnumerable<AdminMenu> adminMenus = workunit.Menu.List(x => x.Online == true,
selector: z => new AdminMenu
{
MenuID = z.MenuID,
Name = z.Name,
Path = z.Path
});
Which throws the exception:
AdminMenu cannot be constructed in a LINQ to Entities query
I have also tried the following approach but it requires to return an IEnumerable<int>:
IEnumerable<AdminMenu> menus = workunit.Menu.List(x => x.Online == true,
selector: z => z.MenuID);
My question is how can I create new instances of my entity class in linq to entities, so not every property is populated.
You cannot pass a selector that creates instances of AdminMenu because that is a type mapped in your Entity Framework context. So Entity Framework wants to be the only one creating instances of that type so it can keep track of changes, return the same instance if already loaded etc... See this question for more info.
So, if you want to return a list of AdminMenu, you won't need to pass a selector (as the dbSet you start querying is already of that type). You would only need to pass a selector when the return type is a different one than the type in the dbSet.
You could have this method: (I have made selector a non-optional parameter as it will be used to define the TResult type. I am also assuming T is a generic parameter defined in the class like public class Repository<T>, so the only generic parameter to add in the method is TResult)
public IEnumerable<TResult> ListProjected<TResult>(Expression<Func<T, TResult>> selector,
Expression<Func<T, bool>> filter = null,
string include = "",
int Taked = 0)
{
IQueryable<T> query = dbSet;
if (filter != null)
query = query.Where(filter);
#region Stringleri İnclude Eder
foreach (var includeProperty in include.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries))
{
query = query.Include(includeProperty);
}
#endregion
if (Taked != 0)
query = query.Take(Taked);
return query.Select(selector).ToList();
}
And a different one when you don't want to project into a different type:
public IEnumerable<T> List(Expression<Func<T, bool>> filter = null,
string include = "",
int Taked = 0)
{
return ListProjected(x => x, filter, include, Taked);
}
You can then use those methods as in:
IEnumerable<AdminMenu> menus = workunit.Menu.List(x => x.Online == true);
IEnumerable<int> menuIds = workunit.Menu.ListProjected(x => x.MenuID,
x => x.Online == true);
Finally, if you only want some columns, you will need to either use an anonymous object or create another class (like a DTO) that only contains the columns you are interested in:
var menuSomeColumnsWithAnonymousObject = workunit.Menu.ListProjected(x => new
{
MenuID = x.MenuID,
Path = x.Path,
Name = x.Name
}, x => x.Id == 1);
var menuSomeColumnsWithDTO = workunit.Menu.ListProjected(x => new AdminMenuDTO
{
MenuID = x.MenuID,
Path = x.Path,
Name = x.Name
}, x => x.Id == 1);
A longer answer than I thought, but I hope it helps!
Iam have run into a little problem with a partial projection.
I have an method on our listboxes so that we can easily load entities into them, however loading full entities is of course not an option.
Meaning i have to do some projection, i have done this here where you can choose which Property to have as the DisplayValue.
Using the code looks like this:
dropCompany.LoadEntityList(Customer.Company, x => x.CompanyName, x => x.IsCompany);
Part of the implementation looks like this:
public void LoadEntityList<T>(T selectedEntity,
System.Linq.Expressions.Expression<Func<T, string>> selector,
System.Linq.Expressions.Expression<Func<T,bool>> where = null)
where T : FlexyBook.Infrastructure.Entity
{
var wCollection = new ObjectWrapperCollection<object>();
IQueryable<T> query = FlexyBook.Repository.DomainService<T>.GetDomainService().GetAll();
if (where != null)
query = query.Where(where);
foreach (var item in query.Select(x =>
new ObjectWrapper<object>()
{
Value = x.ID,
Text = selector.Compile()(x)
})
.ToList().OrderBy(x => x.Text))
{
wCollection.List.Add(item);
}
}
The problem is that this results in the full entity being loaded from the database and then projected.
The issue is also very obvious it is this line "selector.Compile()(x)".
My problem is i have no idea how to allow this partial select to carry on into NHibernate as a projection.
I obviously need the ID selector for identification of the entities and i dont want to change the way the method is called.
Is there anyway to solve this ?
One way do to perform the partial select would be to change your selector expression to be of type:
System.Linq.Expressions.Expression<Func<T, ObjectWrapper<object>>>
Then you would call it like:
dropCompany.LoadEntityList(Customer.Company, x => new ObjectWrapper<object>() { Value = x.ID, Text = x.CompanyName, x => x.IsCompany);
The implementation:
public void LoadEntityList<T>(T selectedEntity, System.Linq.Expressions.Expression<Func<T, ObjectWrapper<object>>> selector,System.Linq.Expressions.Expression<Func<T,bool>> where = null) where T : FlexyBook.Infrastructure.Entity
{
var wCollection = new ObjectWrapperCollection<object>();
IQueryable<T> query = FlexyBook.Repository.DomainService<T>.GetDomainService().GetAll();
if (where != null)
query = query.Where(where);
foreach (var item in query.Select(selector).ToList().OrderBy(x => x.Text))
{
wCollection.List.Add(item);
}
}