Related
I am experimenting with the example CalculatePrice on the Dynamics CRM example page.
And im having a hard time understanding how to get products and bundles in a good manner.
What i wanna try and do is get products from an order with a productstructure attribute and a producttypecode. But it seems whatever i try i get a error The given key was not present in the dictionary.
The query below should look for productID from salesorder based on productID
QueryExpression query = new QueryExpression("salesorderdetail");
query.ColumnSet.AddColumns("quantity", "salesorderispricelocked", "priceperunit", "producttypecode", "_productid_value");
query.Criteria.AddCondition("salesorderid", ConditionOperator.Equal, entity.Id);
QueryExpression query2 = new QueryExpression("product");
query2.ColumnSet.AddColumns("productstructure", "productnumber" , "productid");
query.Criteria.AddCondition("productid", ConditionOperator.Equal, ec.Entities["_productid_value"]);
Then i try to iterate the list of objects to see if they have productstructure and their producttypecode
for (int i = 0; i < ec.Entities.Count; i++)
{
if (ec.Entities[i].GetAttributeValue<int>("producttypecode") == 6)
{ you are a product
if (ec.Entities[i].GetAttributeValue<int>("productstructure") == 3){ you are a bundle
This is the link to the sample code i use:
https://learn.microsoft.com/en-us/dynamics365/customer-engagement/developer/sample-calculate-price-plugin
For starters, the _productid_value notation is the WebAPI's way to access a lookup field. To access the productid using the SDK's late-bound paradigm, use:
myEntity["productid"] or
myEntity.GetAttributeValue<Guid>("productid") or
myEntity.GetAttributeValue<EntityReference>("productid").
Beyond that, since Product is a lookup on the OrderDetail, using a couple LinkEntity objects you could get away with a single query.
I would probably use LINQ and do something like this:
private void getProducts(Guid salesOrderId)
{
using (var context = new Microsoft.Xrm.Sdk.Client.OrganizationServiceContext(svc))
{
var query = from od in context.CreateQuery("salesorderdetail")
join so in context.CreateQuery("salesorder")
on od.GetAttributeValue<Guid>("salesorderid") equals so.GetAttributeValue<Guid>("salesorderid")
join p in context.CreateQuery("product")
on od.GetAttributeValue<Guid>("productid") equals p.GetAttributeValue<Guid>("productid")
where od.GetAttributeValue<Guid>("salesorderid").Equals(salesOrderId)
select new
{
OrderDetailId = od.GetAttributeValue<Guid>("salesorderdetailid"),
ProductId = od.GetAttributeValue<EntityReference>("productid"),
Quantity = od.GetAttributeValue<decimal?>("quantity"),
IsPriceLocked = so.GetAttributeValue<bool?>("ispricelocked"),
PricePerUnit = od.GetAttributeValue<Money>("priceperunit"),
ProductTypeCode = od.GetAttributeValue<OptionSetValue>("producttypecode"),
ProductStructure = p.GetAttributeValue<OptionSetValue>("productstructure"),
ProductNumber = p.GetAttributeValue<string>("productnumber")
};
var results = query.ToList();
var products = results.Where(e => e.ProductStructure.Value == 6).ToList();
var bundles = results.Where(e => e.ProductStructure.Value == 3).ToList();
}
}
Please note that local variables results, products, and bundles are an anonymous type. You can loop through and access the properties of each object, but there's also a strong chance you'd want to cast them into instances of a real class.
I have 3 tables linked by Foreign Keys: ChangeSet, ObjectChanges, PropertyChanges. These tables have 1-To-Many Relationships with each other. I need to join and project and flatten the results into an anonymous type.
We use Entity Framework at the data layer and I essentially need to make the following Query with linq.
select c.Id as ChangeSetId,
c.Timestamp,
c.Author_Id,
u.Name as [User],
o.id as ObjectChangeId,
o.TypeName,
o.ObjectReference as EntityId,
o.DisplayName,
p.Id as PropertyChangeId,
p.PropertyName,
p.ChangeType,
p.OriginalValue,
p.Value
from ChangeSets c
inner join ObjectChanges o
on c.Id = o.ChangeSetId
left join PropertyChanges p
on p.ObjectChangeId = o.Id
inner join Users u
on u.Id = c.Author_Id
order by c.id desc
The Method in question however looks like this:
GetAllWhereExpression(Expression<Func<ChangeSet, bool>> expression)
The expression in this case is likely to be a Where o.EntityId = [Some Value] and c.TimeStamp > X and < Y.
I got very close I felt in linq with the following but couldn't figure out how to inject the expression: (The .GetRepository().Entities is basically DbSet)
var foo = from c in _uow.GetRepository<ChangeSet>().Entities
join o in _uow.GetRepository<ObjectChange>().Entities on c.Id equals o.ChangeSetId
join p in _uow.GetRepository<PropertyChange>().Entities on o.Id equals p.ObjectChangeId
where expression // This Line Not Valid
select new
{
ChangeSetId = c.Id,
Timestamp = c.Timestamp,
User = c.User.DisplayName,
EntityType = o.TypeName,
EntityValue = o.DisplayName,
Property = p.PropertyName,
OldValue = p.OriginalValue,
NewValue = p.Value
};
I'd prefer to use Lambda syntax but I can't figure out how to construct it. I know I need SelectMany to project and flatten the results but I can't figure out how to use them within the anonymous type for the subcollections:
var queryable = _uow.GetRepository<ChangeSet>().Entities // This is basically the DbSet<ChangeSet>()
.Where(expression)
.SelectMany(c => new
{
ChangeSetId = c.Id,
Timestamp = c.Timestamp,
User = c.User.DisplayName,
EntityType = c.ObjectChanges.SelectMany(o => o.TypeName), //Doesn't work, turns string into char array
//PropertyName = c. this would be a 1 to many on the entity
}
)
How do I craft the linq to produce basically the same results as the sql query?
Here is how it may look like in method syntax.
_uow.GetRepository<ChangeSet>().Entities
.Where(expression)
.Join(_uow.GetRepository<ObjectChanges>().Entities, cs => cs.Id, oc => oc.ChangeSetId,
(cs, oc) => new { cs, oc })
.Join(_uow.GetRepository<PropertyChanges>().Entities, outer => outer.oc.Id, pc => pc.ObjectChangeId,
(outer, pc) => new { cs = outer.cs, oc = outer.cs, pc })
.Join(_uow.GetRepository<User>().Entities, outer => outer.cs.Author_Id, u => u.Id,
(outer, u) => new {
ChangeSetId = outer.cs.Id,
Timestamp = outer.cs.Timestamp,
User = u.DisplayName,
EntityType = outer.oc.TypeName,
EntityValue = outer.oc.DisplayName,
Property = outer.pc.PropertyName,
OldValue = outer.pc.OriginalValue,
NewValue = outer.pc.Value
})
Please follow the given demonstration, Four entities joined by LINQ extension methods.
N.B: .Join use for Inner Join
var foo=_uow.GetRepository<ChangeSet>().Entities
.Join(_uow.GetRepository<ObjectChange>().Entities,
c=>c.Id,
o=>o.ChangeSetId,
(c,o)=>new{ChangeSet=c,ObjectChange=o})
.Join(_uow.GetRepository<PropertyChange>().Entities,
o=>o.ObjectChange.Id,
p=>p.ObjectChangeId,
(o,p)=>new{o.ChangeSet,o.ObjectChange,PropertyChange=p})
.Join(_uow.GetRepository<Users>().Entities,
c=>c.ChangeSet.Author_Id,
u=>u.Id,
(c,u)=>new{c.ChangeSet,c.ObjectChange,c.PropertyChange,User=u})
.Select(x=>new
{
ChangeSetId=x.ChangeSet.Id,
x.ChangeSet.Timestamp,
x.ChangeSet.Author_Id,
User=x.User.Name,
ObjectChangeId=x.ObjectChange.id,
x.ObjectChange.TypeName,
EntityId=x.ObjectChange.ObjectReference,
x.ObjectChange.DisplayName,
PropertyChangeId=x.PropertyChange.Id,
x.PropertyChange.PropertyName,
x.PropertyChange.ChangeType,
x.PropertyChange.OriginalValue,
x.PropertyChange.Value
}).OrderByDescending(x=>x.ChangeSetId).ToList() //ChangeSetId=c.Id
To keep it simple imagine each table has a column named Description and we want to retrieve a flat result like this:
-------------------------------------------------------------------------------------
ChangeSetDescription | ObjectChangeDescription | PropertyChangeDescription |
-------------------------------------------------------------------------------------
Short Answer
You do not need to do a join because EF will figure out the joining for you based on your data model. You can just do this. Imagine ctx is an instance of a class which derives DbContext :
var query = ctx.ChangeSets
.SelectMany(x => x.ObjectChanges, (a, oc) => new
{
ChangeSetDescription = a.Description,
ObjectChangeDescription = oc.Description
,
Pcs = oc.PropertyChanges
})
.SelectMany(x => x.Pcs, (a, pc) => new
{
ChangeSetDescription = a.ChangeSetDescription,
ObjectChangeDescription = a.ObjectChangeDescription,
PropertyChangeDesription = pc.Description
});
var result = query.ToList();
Long Answer
Why do I not need the join?
When you create your EF model, regardless of whether you are using code first, database first or a hybrid approach, if you create the relationships correctly then EF will figure out the joins for you. In a database query, we need to tell the database engine how to join, but in EF we do not need to do this. EF will figure out the relationship based on your model and do the joins using the navigation properties. The only time we should be using joins is if we are joining on some arbitrary property. In other words, we should seldom use joins in EF queries.
Your EF queries should not be written as a direct translation of SQL queries. Take advantage of EF and let it do the work for you.
Can you explain the query in the short answer?
It is the SelectMany method which will throw readers off a little. But it is pretty easy. Let's use a simple class as example:
public class Student
{
public string Name { get; private set; }
public IEnumerable<string> Courses { get; private set; }
public Student(string name, params string[] courses)
{
this.Name = name;
this.Courses = courses;
}
}
Let's create 2 students and add them to a generic list:
var s1 = new Student("George", "Math");
var s2 = new Student("Jerry", "English", "History");
var students = new List<Student> { s1, s2 };
Now let's imagine we need to get the name of the students and the classes they are taking as flat result set. In other words, we do not want Jerry to have a list with 2 subjects but instead we want 2 records for Jerry. Naturally, we will think of SelectMany and do this:
var courses = students.SelectMany(x => new { Name = x.Name, Courses = x.Courses });
But that will not work and cause a compilation error:
The type arguments for method 'Enumerable.SelectMany<TSource, TResult>
(IEnumerable<TSource>, Func<TSource, IEnumerable<TResult>>)'
cannot be inferred >from the usage. Try specifying the type arguments explicitly.
The SelectMany method takes a Func which will return IEnumerable<TResult> NOT TResult so that will not work. Luckily, it has an overload that we can use. But the overload has a, well, intimidating signature:
public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>
(
this IEnumerable<TSource> source,
Func<TSource, IEnumerable<TCollection>> collectionSelector,
Func<TSource, TCollection, TResult> resultSelector);
Basically what that is saying is the first Func should return an IEnumerable<TCollection> and the second will receive the TSource and TCollection as input and we can return a TResult as output. Ok so let's do that:
var courses = students.SelectMany(x => x.Courses,
(a, b) => new { Name = a.Name, Course = b });
So in the above line, a is the source which is our students list and b is the string yielded to us one by one from the first Func which we said will return x.Courses. The TResult we are returning is an anonymous type.
Hopefully this example will clarify what is going on in the short answer.
Or what you can do is, since your where expression is for changeSet table, you can filter it when you select from it within your linq
var foo = from c in _uow.GetRepository<ChangeSet>().Entities.Where(expression) //Notice lambda exp here
join o in _uow.GetRepository<ObjectChange>().Entities on c.Id equals o.ChangeSetId
join p in _uow.GetRepository<PropertyChange>().Entities on o.Id equals p.ObjectChangeId
select new
{
ChangeSetId = c.Id,
Timestamp = c.Timestamp,
User = c.User.DisplayName,
EntityType = o.TypeName,
EntityValue = o.DisplayName,
Property = p.PropertyName,
OldValue = p.OriginalValue,
NewValue = p.Value
};
I have searching for a while now, but couldn't find how to query from public view. For example, I have predefined public view called Active Accounts and I want data from it.
So far I only know this way, but that not include any views:
using (var xrm = new XrmServiceContext("Xrm"))
{
var activeAccounts = from a in xrm.AccountSet
where a.StateCode == 0
select new { a.Id, a.Name };
// TODO ...
}
But I would like to do it like this (not working, ActiveAccountsView not exist, it's pseudo):
using (var xrm = new XrmServiceContext("Xrm"))
{
var activeAccounts = from a in xrm.ActiveAccountsView
select new { a.Id, a.Name };
// TODO ...
}
Is this even possible?
The query definitions of public views are stored in the savedquery entity, that can be queried using common techniques.
Out-of-the-box views are stored with a fixed ID, so querying Active Accounts on the OrganizationServiceContext object could be done in the following way:
private static IEnumerable<Entity> GetActiveAccounts(OrganizationServiceContext serviceContext)
{
string fetchXml = serviceContext
.CreateQuery("savedquery")
.Where(sq =>
sq.GetAttributeValue<Guid>("savedqueryid") == new Guid("00000000-0000-0000-00AA-000010001002"))
.Select(sq => sq.GetAttributeValue<string>("fetchxml"))
.First();
var request = new RetrieveMultipleRequest
{
Query = new FetchExpression(fetchXml)
};
var response = (RetrieveMultipleResponse) serviceContext.Execute(request);
return response.EntityCollection.Entities;
}
It is not possible to use LINQ here. LINQ relies on the QueryExpression class, but does not implement all its capabilities (OUTER JOIN is a painful omission for example). So, while it is possible to convert a LINQ query to a QueryExpression, the other way around is not.
Paging can be applied by editing the Fetch XML string, but if that is too much hassle, you can also consider to convert the Fetch XML to a QueryExpression and apply paging on that object:
private IEnumerable<Entity> GetActiveAccounts(int pageNumber)
{
string fetchXml = _serviceContext
.CreateQuery("savedquery")
.Where(sq =>
sq.GetAttributeValue<Guid>("savedqueryid") == new Guid("00000000-0000-0000-00AA-000010001002"))
.Select(sq => sq.GetAttributeValue<string>("fetchxml"))
.First();
var conversionRequest = new FetchXmlToQueryExpressionRequest
{
FetchXml = fetchXml
};
var response = (FetchXmlToQueryExpressionResponse)_serviceContext.Execute(conversionRequest);
response.Query.PageInfo = new PagingInfo { Count = 1, PageNumber = pageNumber };
var queryRequest = new RetrieveMultipleRequest
{
Query = response.Query
};
var result = (RetrieveMultipleResponse) _serviceContext.Execute(queryRequest);
return result.EntityCollection.Entities;
}
Additional advantage of the QueryExpression vs. Fetch XML is that it is processed in a bit more efficient way.
The very same can be done with user defined views; these views are stored in the userquery entity. The only difference here is you cannot rely on a fixed view ID. Instead you would need to filter your query on querytype, name, returnedtypecode, ownerid and/or other criteria.
Dynamics CRM also has an OrganizationRequest that allows you to execute the savedquery immediately. However, it returns its result as a resultset XML string, so you would still need to deserialize the response. (A nice example can be found here.) Also, I am not sure if it is possible to limit the result set to a specific page when using the ExecuteByIdSavedQueryRequest:
var request = new ExecuteByIdSavedQueryRequest
{
EntityId = new Guid("00000000-0000-0000-00AA-000010001002")
};
var response = (ExecuteByIdSavedQueryResponse)serviceContext.Execute(request);
string resultset = response.String;
given a list of ids, I can query all relevant rows by:
context.Table.Where(q => listOfIds.Contains(q.Id));
But how do you achieve the same functionality when the Table has a composite key?
This is a nasty problem for which I don't know any elegant solution.
Suppose you have these key combinations, and you only want to select the marked ones (*).
Id1 Id2
--- ---
1 2 *
1 3
1 6
2 2 *
2 3 *
... (many more)
How to do this is a way that Entity Framework is happy? Let's look at some possible solutions and see if they're any good.
Solution 1: Join (or Contains) with pairs
The best solution would be to create a list of the pairs you want, for instance Tuples, (List<Tuple<int,int>>) and join the database data with this list:
from entity in db.Table // db is a DbContext
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
In LINQ to objects this would be perfect, but, too bad, EF will throw an exception like
Unable to create a constant value of type 'System.Tuple`2 (...) Only primitive types or enumeration types are supported in this context.
which is a rather clumsy way to tell you that it can't translate this statement into SQL, because Tuples is not a list of primitive values (like int or string). For the same reason a similar statement using Contains (or any other LINQ statement) would fail.
Solution 2: In-memory
Of course we could turn the problem into simple LINQ to objects like so:
from entity in db.Table.AsEnumerable() // fetch db.Table into memory first
join pair Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
Needless to say that this is not a good solution. db.Table could contain millions of records.
Solution 3: Two Contains statements (incorrect)
So let's offer EF two lists of primitive values, [1,2] for Id1 and [2,3] for Id2. We don't want to use join, so let's use Contains:
from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity
But now the results also contains entity {1,3}! Well, of course, this entity perfectly matches the two predicates. But let's keep in mind that we're getting closer. In stead of pulling millions of entities into memory, we now only get four of them.
Solution 4: One Contains with computed values
Solution 3 failed because the two separate Contains statements don't only filter the combinations of their values. What if we create a list of combinations first and try to match these combinations? We know from solution 1 that this list should contain primitive values. For instance:
var computed = ids1.Zip(ids2, (i1,i2) => i1 * i2); // [2,6]
and the LINQ statement:
from entity in db.Table
where computed.Contains(entity.Id1 * entity.Id2)
select entity
There are some problems with this approach. First, you'll see that this also returns entity {1,6}. The combination function (a*b) does not produce values that uniquely identify a pair in the database. Now we could create a list of strings like ["Id1=1,Id2=2","Id1=2,Id2=3]" and do
from entity in db.Table
where computed.Contains("Id1=" + entity.Id1 + "," + "Id2=" + entity.Id2)
select entity
(This would work in EF6, not in earlier versions).
This is getting pretty messy. But a more important problem is that this solution is not sargable, which means: it bypasses any database indexes on Id1 and Id2 that could have been used otherwise. This will perform very very poorly.
Solution 5: Best of 2 and 3
So the most viable solution I can think of is a combination of Contains and a join in memory: First do the contains statement as in solution 3. Remember, it got us very close to what we wanted. Then refine the query result by joining the result as an in-memory list:
var rawSelection = from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity;
var refined = from entity in rawSelection.AsEnumerable()
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity;
It's not elegant, messy all the same maybe, but so far it's the only scalable1 solution to this problem I found, and applied in my own code.
Solution 6: Build a query with OR clauses
Using a Predicate builder like Linqkit or alternatives, you can build a query that contains an OR clause for each element in the list of combinations. This could be a viable option for really short lists. With a couple of hundreds of elements, the query will start performing very poorly. So I don't consider this a good solution unless you can be 100% sure that there will always be a small number of elements. One elaboration of this option can be found here.
Solution 7: Unions
There's also a solution using UNIONs that I posted later here.
1As far as the Contains statement is scalable: Scalable Contains method for LINQ against a SQL backend
Solution for Entity Framework Core with SQL Server
🎉 NEW! QueryableValues EF6 Edition has arrived!
The following solution makes use of QueryableValues. This is a library that I wrote to primarily solve the problem of query plan cache pollution in SQL Server caused by queries that compose local values using the Contains LINQ method. It also allows you to compose values of complex types in your queries in a performant way, which will achieve what's being asked in this question.
First you will need to install and set up the library, after doing that you can use any of the following patterns that will allow you to query your entities using a composite key:
// Required to make the AsQueryableValues method available on the DbContext.
using BlazarTech.QueryableValues;
// Local data that will be used to query by the composite key
// of the fictitious OrderProduct table.
var values = new[]
{
new { OrderId = 1, ProductId = 10 },
new { OrderId = 2, ProductId = 20 },
new { OrderId = 3, ProductId = 30 }
};
// Optional helper variable (needed by the second example due to CS0854)
var queryableValues = dbContext.AsQueryableValues(values);
// Example 1 - Using a Join (preferred).
var example1Results = dbContext
.OrderProduct
.Join(
queryableValues,
e => new { e.OrderId, e.ProductId },
v => new { v.OrderId, v.ProductId },
(e, v) => e
)
.ToList();
// Example 2 - Using Any (similar behavior as Contains).
var example2Results = dbContext
.OrderProduct
.Where(e => queryableValues
.Where(v =>
v.OrderId == e.OrderId &&
v.ProductId == e.ProductId
)
.Any()
)
.ToList();
Useful Links
Nuget Package
GitHub Repository
Benchmarks
QueryableValues is distributed under the MIT license.
You can use Union for each composite primary key:
var compositeKeys = new List<CK>
{
new CK { id1 = 1, id2 = 2 },
new CK { id1 = 1, id2 = 3 },
new CK { id1 = 2, id2 = 4 }
};
IQuerable<CK> query = null;
foreach(var ck in compositeKeys)
{
var temp = context.Table.Where(x => x.id1 == ck.id1 && x.id2 == ck.id2);
query = query == null ? temp : query.Union(temp);
}
var result = query.ToList();
You can create a collection of strings with both keys like this (I am assuming that your keys are int type):
var id1id2Strings = listOfIds.Select(p => p.Id1+ "-" + p.Id2);
Then you can just use "Contains" on your db:
using (dbEntities context = new dbEntities())
{
var rec = await context.Table1.Where(entity => id1id2Strings .Contains(entity.Id1+ "-" + entity.Id2));
return rec.ToList();
}
You need a set of objects representing the keys you want to query.
class Key
{
int Id1 {get;set;}
int Id2 {get;set;}
If you have two lists and you simply check that each value appears in their respective list then you are getting the cartesian product of the lists - which is likely not what you want. Instead you need to query the specific combinations required
List<Key> keys = // get keys;
context.Table.Where(q => keys.Any(k => k.Id1 == q.Id1 && k.Id2 == q.Id2));
I'm not completely sure that this is valid use of Entity Framework; you may have issues with sending the Key type to the database. If that happens then you can be creative:
var composites = keys.Select(k => p1 * k.Id1 + p2 * k.Id2).ToList();
context.Table.Where(q => composites.Contains(p1 * q.Id1 + p2 * q.Id2));
You can create an isomorphic function (prime numbers are good for this), something like a hashcode, which you can use to compare the pair of values. As long as the multiplicative factors are co-prime this pattern will be isomorphic (one-to-one) - i.e. the result of p1*Id1 + p2*Id2 will uniquely identify the values of Id1 and Id2 as long as the prime numbers are correctly chosen.
But then you end up in a situation where you're implementing complex concepts and someone is going to have to support this. Probably better to write a stored procedure which takes the valid key objects.
Ran into this issue as well and needed a solution that both did not perform a table scan and also provided exact matches.
This can be achieved by combining Solution 3 and Solution 4 from Gert Arnold's Answer
var firstIds = results.Select(r => r.FirstId);
var secondIds = results.Select(r => r.SecondId);
var compositeIds = results.Select(r => $"{r.FirstId}:{r.SecondId}");
var query = from e in dbContext.Table
//first check the indexes to avoid a table scan
where firstIds.Contains(e.FirstId) && secondIds.Contains(e.SecondId))
//then compare the compositeId for an exact match
//ToString() must be called unless using EF Core 5+
where compositeIds.Contains(e.FirstId.ToString() + ":" + e.SecondId.ToString()))
select e;
var entities = await query.ToListAsync();
For EF Core I use a slightly modified version of the bucketized IN method by EricEJ to map composite keys as tuples. It performs pretty well for small sets of data.
Sample usage
List<(int Id, int Id2)> listOfIds = ...
context.Table.In(listOfIds, q => q.Id, q => q.Id2);
Implementation
public static IQueryable<TQuery> In<TKey1, TKey2, TQuery>(
this IQueryable<TQuery> queryable,
IEnumerable<(TKey1, TKey2)> values,
Expression<Func<TQuery, TKey1>> key1Selector,
Expression<Func<TQuery, TKey2>> key2Selector)
{
if (values is null)
{
throw new ArgumentNullException(nameof(values));
}
if (key1Selector is null)
{
throw new ArgumentNullException(nameof(key1Selector));
}
if (key2Selector is null)
{
throw new ArgumentNullException(nameof(key2Selector));
}
if (!values.Any())
{
return queryable.Take(0);
}
var distinctValues = Bucketize(values);
if (distinctValues.Length > 1024)
{
throw new ArgumentException("Too many parameters for SQL Server, reduce the number of parameters", nameof(values));
}
var predicates = distinctValues
.Select(v =>
{
// Create an expression that captures the variable so EF can turn this into a parameterized SQL query
Expression<Func<TKey1>> value1AsExpression = () => v.Item1;
Expression<Func<TKey2>> value2AsExpression = () => v.Item2;
var firstEqual = Expression.Equal(key1Selector.Body, value1AsExpression.Body);
var visitor = new ReplaceParameterVisitor(key2Selector.Parameters[0], key1Selector.Parameters[0]);
var secondEqual = Expression.Equal(visitor.Visit(key2Selector.Body), value2AsExpression.Body);
return Expression.AndAlso(firstEqual, secondEqual);
})
.ToList();
while (predicates.Count > 1)
{
predicates = PairWise(predicates).Select(p => Expression.OrElse(p.Item1, p.Item2)).ToList();
}
var body = predicates.Single();
var clause = Expression.Lambda<Func<TQuery, bool>>(body, key1Selector.Parameters[0]);
return queryable.Where(clause);
}
class ReplaceParameterVisitor : ExpressionVisitor
{
private ParameterExpression _oldParameter;
private ParameterExpression _newParameter;
public ReplaceParameterVisitor(ParameterExpression oldParameter, ParameterExpression newParameter)
{
_oldParameter = oldParameter;
_newParameter = newParameter;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (ReferenceEquals(node, _oldParameter))
return _newParameter;
return base.VisitParameter(node);
}
}
/// <summary>
/// Break a list of items tuples of pairs.
/// </summary>
private static IEnumerable<(T, T)> PairWise<T>(this IEnumerable<T> source)
{
var sourceEnumerator = source.GetEnumerator();
while (sourceEnumerator.MoveNext())
{
var a = sourceEnumerator.Current;
sourceEnumerator.MoveNext();
var b = sourceEnumerator.Current;
yield return (a, b);
}
}
private static TKey[] Bucketize<TKey>(IEnumerable<TKey> values)
{
var distinctValueList = values.Distinct().ToList();
// Calculate bucket size as 1,2,4,8,16,32,64,...
var bucket = 1;
while (distinctValueList.Count > bucket)
{
bucket *= 2;
}
// Fill all slots.
var lastValue = distinctValueList.Last();
for (var index = distinctValueList.Count; index < bucket; index++)
{
distinctValueList.Add(lastValue);
}
var distinctValues = distinctValueList.ToArray();
return distinctValues;
}
In the absence of a general solution, I think there are two things to consider:
Avoid multi-column primary keys (will make unit testing easier too).
But if you have to, chances are that one of them will reduce the
query result size to O(n) where n is the size of the ideal query
result. From here, its Solution 5 from Gerd Arnold above.
For example, the problem leading me to this question was querying order lines, where the key is order id + order line number + order type, and the source had the order type being implicit. That is, the order type was a constant, order ID would reduce the query set to order lines of relevant orders, and there would usually be 5 or less of these per order.
To rephrase: If you have a composite key, changes are that one of them have very few duplicates. Apply Solution 5 from above with that.
I tried this solution and it worked with me and the output query was perfect without any parameters
using LinqKit; // nuget
var customField_Ids = customFields?.Select(t => new CustomFieldKey { Id = t.Id, TicketId = t.TicketId }).ToList();
var uniqueIds1 = customField_Ids.Select(cf => cf.Id).Distinct().ToList();
var uniqueIds2 = customField_Ids.Select(cf => cf.TicketId).Distinct().ToList();
var predicate = PredicateBuilder.New<CustomFieldKey>(false); //LinqKit
var lambdas = new List<Expression<Func<CustomFieldKey, bool>>>();
foreach (var cfKey in customField_Ids)
{
var id = uniqueIds1.Where(uid => uid == cfKey.Id).Take(1).ToList();
var ticketId = uniqueIds2.Where(uid => uid == cfKey.TicketId).Take(1).ToList();
lambdas.Add(t => id.Contains(t.Id) && ticketId.Contains(t.TicketId));
}
predicate = AggregateExtensions.AggregateBalanced(lambdas.ToArray(), (expr1, expr2) =>
{
var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
return Expression.Lambda<Func<CustomFieldKey, bool>>
(Expression.OrElse(expr1.Body, invokedExpr), expr1.Parameters);
});
var modifiedCustomField_Ids = repository.GetTable<CustomFieldLocal>()
.Select(cf => new CustomFieldKey() { Id = cf.Id, TicketId = cf.TicketId }).Where(predicate).ToArray();
I ended up writing a helper for this problem that relies on System.Linq.Dynamic.Core;
Its a lot of code and don't have time to refactor at the moment but input / suggestions appreciated.
public static IQueryable<TEntity> WhereIsOneOf<TEntity, TSource>(this IQueryable<TEntity> dbSet,
IEnumerable<TSource> source,
Expression<Func<TEntity, TSource,bool>> predicate) where TEntity : class
{
var (where, pDict) = GetEntityPredicate(predicate, source);
return dbSet.Where(where, pDict);
(string WhereStr, IDictionary<string, object> paramDict) GetEntityPredicate(Expression<Func<TEntity, TSource, bool>> func, IEnumerable<TSource> source)
{
var firstP = func.Parameters[0];
var binaryExpressions = RecurseBinaryExpressions((BinaryExpression)func.Body);
var i = 0;
var paramDict = new Dictionary<string, object>();
var res = new List<string>();
foreach (var sourceItem in source)
{
var innerRes = new List<string>();
foreach (var bExp in binaryExpressions)
{
var emp = ToEMemberPredicate(firstP, bExp);
var val = emp.GetKeyValue(sourceItem);
var pName = $"#{i++}";
paramDict.Add(pName, val);
var str = $"{emp.EntityMemberName} {emp.SQLOperator} {pName}";
innerRes.Add(str);
}
res.Add( "(" + string.Join(" and ", innerRes) + ")");
}
var sRes = string.Join(" || ", res);
return (sRes, paramDict);
}
EMemberPredicate ToEMemberPredicate(ParameterExpression firstP, BinaryExpression bExp)
{
var lMember = (MemberExpression)bExp.Left;
var rMember = (MemberExpression)bExp.Right;
var entityMember = lMember.Expression == firstP ? lMember : rMember;
var keyMember = entityMember == lMember ? rMember : lMember;
return new EMemberPredicate(entityMember, keyMember, bExp.NodeType);
}
List<BinaryExpression> RecurseBinaryExpressions(BinaryExpression e, List<BinaryExpression> runningList = null)
{
if (runningList == null) runningList = new List<BinaryExpression>();
if (e.Left is BinaryExpression lbe)
{
var additions = RecurseBinaryExpressions(lbe);
runningList.AddRange(additions);
}
if (e.Right is BinaryExpression rbe)
{
var additions = RecurseBinaryExpressions(rbe);
runningList.AddRange(additions);
}
if (e.Left is MemberExpression && e.Right is MemberExpression)
{
runningList.Add(e);
}
return runningList;
}
}
Helper class:
public class EMemberPredicate
{
public readonly MemberExpression EntityMember;
public readonly MemberExpression KeyMember;
public readonly PropertyInfo KeyMemberPropInfo;
public readonly string EntityMemberName;
public readonly string SQLOperator;
public EMemberPredicate(MemberExpression entityMember, MemberExpression keyMember, ExpressionType eType)
{
EntityMember = entityMember;
KeyMember = keyMember;
KeyMemberPropInfo = (PropertyInfo)keyMember.Member;
EntityMemberName = entityMember.Member.Name;
SQLOperator = BinaryExpressionToMSSQLOperator(eType);
}
public object GetKeyValue(object o)
{
return KeyMemberPropInfo.GetValue(o, null);
}
private string BinaryExpressionToMSSQLOperator(ExpressionType eType)
{
switch (eType)
{
case ExpressionType.Equal:
return "==";
case ExpressionType.GreaterThan:
return ">";
case ExpressionType.GreaterThanOrEqual:
return ">=";
case ExpressionType.LessThan:
return "<";
case ExpressionType.LessThanOrEqual:
return "<=";
case ExpressionType.NotEqual:
return "<>";
default:
throw new ArgumentException($"{eType} is not a handled Expression Type.");
}
}
}
Use Like so:
// This can be a Tuple or whatever.. If Tuple, then y below would be .Item1, etc.
// This data structure is up to you but is what I use.
[FromBody] List<CustomerAddressPk> cKeys
var res = await dbCtx.CustomerAddress
.WhereIsOneOf(cKeys, (x, y) => y.CustomerId == x.CustomerId
&& x.AddressId == y.AddressId)
.ToListAsync();
Hope this helps others.
in Case of composite key you can use another idlist and add a condition for that in your code
context.Table.Where(q => listOfIds.Contains(q.Id) && listOfIds2.Contains(q.Id2));
or you can use one another trick create a list of your keys by adding them
listofid.add(id+id1+......)
context.Table.Where(q => listOfIds.Contains(q.Id+q.id1+.......));
I tried this on EF Core 5.0.3 with the Postgres provider.
context.Table
.Select(entity => new
{
Entity = entity,
CompositeKey = entity.Id1 + entity.Id2,
})
.Where(x => compositeKeys.Contains(x.CompositeKey))
.Select(x => x.Entity);
This produced SQL like:
SELECT *
FROM table AS t
WHERE t.Id1 + t.Id2 IN (#__compositeKeys_0)),
Caveats
this should only be used where the combination of Id1 and Id2 will always produce a unique result (e.g., they're both UUIDs)
this cannot use indexes, though you could save the composite key to the db with an index
I have a database table that contains an nvarchar column like this:
1|12.6|18|19
I have a Business Object that has a Decimal[] property.
My LINQ Query looks like this:
var temp = from r in db.SomeTable select new BusinessObject {
// Other BusinessObject Properties snipped as they are straight 1:1
MeterValues = r.MeterValues.Split('|').Select(Decimal.Parse).ToArray()
};
var result = temp.ToArray();
This throws an NotSupportedException: Method 'System.String[] Split(Char[])' has no supported translation to SQL.
That kinda sucks :) Is there any way I can do this without having to add a string property to the business object or selecting an anonymous type and then iterating through it?
My current "solution" is:
var temp = from r in db.SomeTable select new {
mv = r.MeterValues,
bo = new BusinessObject { // all the other fields }
};
var result = new List<BusinessObject>();
foreach(var t in temp) {
var bo = t.bo;
bo.MeterValues = t.mv.Split('|').Select(Decimal.Parse).ToArray();
result.Add(bo);
}
return result.ToArray(); // The Method returns BusinessObject[]
That's kinda ugly though, with that temporary list.
I've tried adding a let mv = r.MeterValues.Split('|').Select(Decimal.Parse).ToArray() but that essentially leads to the same NotSupportedException.
This is .net 3.5SP1 if that matters.
You need to force the select clause to run on the client by calling .AsEnumerable() first:
var result = db.SomeTable.AsEnumerable().Select(r => new BusinessObject {
...
MeterValues = r.MeterValues.Split('|').Select(Decimal.Parse).ToArray()
}).ToList();
You can't use split, but in this scenario you can do the following:
// Database value is 1|12.6|18|19
string valueToFind = "19";
var temp = from r in db.SomeTable.Where(r => ("|" + r.MeterValues + "|").Contains("|" + valueToFind + "|"));
This code adds outer pipes (|) to the database value on the fly inside the query so you can do start, middle, and end value matches on the string.
For example, the above code looks for "|19|" inside "|1|12.6|18|19|", which is found and valid. This will work for any other valueToFind.
You don't need to use a temporary list:
var query = from r in db.SomeTable
select new
{
r.Id,
r.Name,
r.MeterValues,
...
};
var temp = from x in query.AsEnumerable()
select new BusinessObject
{
Id = x.Id,
Name = x.Name,
MeterValues = x.mv.Split('|').Select(Decimal.Parse).ToArray(),
...
};
return temp.ToArray();
Unfortunately its the IQueryable you are using (Linq to SQL) that is not supporting the Split function.
You are really only left with the IEnumerable (Linq to Objects) support for it in this case. You second code snippet is what you need to do, or something like...
var temp = (from r in db.SomeTable select new {
mv = r.MeterValues,
bo = new BusinessObject { // all the other fields }
}).AsEnumerable().Select(blah, blah) ;