We Keep Coding

EFCore join values

I'm trying to join a big table to a small list of data pairs using EFCore 2.1.1. I want this join to happen server-side, rather than trying to download the whole table, e.g translating to something like:
SELECT a.*
FROM Groups AS a
INNER JOIN (VALUES (1, 'admins'), (2, 'support'), (1, 'admins')) AS b(organization_id, name)
ON a.organization_id = b.organization_id AND a.name = b.name;
or something equivalent (e.g. using common table expressions). Is this possible? If so, how? Passing a list of objects to a LINQ .join seems to always get handled client-side.
Due to massive testing debt and the EFCore 3 breaking change on client-side evaluation, upgrading is not an option for us at this time (but answers relevant to newer versions may help us push management)

If you expect that EF Core 3.x can support this, you are wrong. If you plan to upgrade your application, better think about EF Core 6 and .net 6.
Anyway I know several options:
With extension method FilterByItems or similar
var items = ...
var query = context.Groups
.FilterByItems(items, (q, b) => q.organization_id == b.organization_id && q.name == i.name, true);
With third party extension inq2db.EntityFrameworkCore version 2.x, note that I'm one of the creators. It will generate exactly the same SQL as in question.
var items = ...
var query =
from g in context.Groups
join b in items on new { g.organization_id, g.name } equals new { b.organization_id, b.name }
select g;
var result = query.ToLinqToDB().ToList();

You could solve this problem with a Table Value Parameter.
First define a database type;
IF TYPE_ID(N'[IdName]') IS NULL
CREATE TYPE [IdName] AS TABLE (
[Id] int NOT NULL
[Name] nvarchar(max) NOT NULL
)
Then you can build an SqlParameter from an IEnumerable;
public class IdName
{
public int Id { get; set; }
[Required]
public string Name { get; set; }
public static SqlParameter ToParameter(IEnumerable<IdName> values)
{
var meta = new SqlMetaData[]{
new SqlMetaData(nameof(Id), SqlDbType.Int),
new SqlMetaData(nameof(Name), SqlDbType.NVarChar, int.MaxValue)
};
return new SqlParameter()
{
TypeName = nameof(IdName),
SqlDbType = SqlDbType.Structured,
Value = values.Select(v => {
var record = new SqlDataRecord(meta);
record.SetInt32(0, v.Id);
record.SetString(1, v.Name);
return record;
})
};
}
}
Then define an EF Core query type, and you can turn that SqlParameter into an IQueryable;
public void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<IdName>(e =>
{
e.HasNoKey();
e.ToView(nameof(IdName));
});
}
public static IQueryable<IdName> ToQueryable(DbContext db, IEnumerable<IdName> values)
=> db.Set<IdName>().FromSqlInterpolated($"select * from {ToParameter(values)}");
And now you can use that IQueryable in a Linq join.

Linq to entities - SQL Query - Where list contains object with 2 properties (or more)

Having the following example:
var myIds = db.Table1.Where(x=>x.Prop2 == myFilter).Select(x=>x.Id).ToList();
var results = db.Table2.Where(x=> myIds.Contains(x.T1)).ToList();
This part is straight forward.
However, now I am facing a "slight" change where my "filter list" has 2 properties instead of only one:
// NOTE: for stackoverflow simplification I use a basic query to
// get my "myCombinationObject".
// In reality this is a much more complex case,
// but the end result is a LIST of objects with two properties.
var myCombinationObject = db.Table3.Where(x=>x.Prop3 == myFilter)
.Select(x=> new {
Id1 = x.T1,
Id2 = x.T2
}).ToList();
var myCombinationObjectId1s = myCombinationObject.Select(x=>xId1).ToList();
var myCombinationObjectId2s = myCombinationObject.Select(x=>xId2).ToList();
// step#1 - DB SQL part
var resultsRaw = db.Tables.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.ToList();
// step#2 - Now in memory side - where I make the final combination filter.
var resultsFiltered = resultsRaw.Where( x=>
myCombinationObject.Contains(
new {Id1 = x.Prop1, Id2 = x.Prop2 }
).ToList();
My question: is it even possible to merge the step#2 in the step#1 (query in linq to entities) ?

I've managed once to do what you want, however it is pretty hard and requires changing entity model a bit. You need an entity to map type
new {Id1 = x.Prop1, Id2 = x.Prop2 }
So you need enity having 2 properties - Id1 and Id2. If you have one - great, if not then add such entity to your model:
public class CombinationObjectTable
{
public virtual Guid Id1 { get; set; }
public virtual Guid Id2 { get; set; }
}
Add it to your model:
public DbSet<CombinationObjectTable> CombinationObjectTable { get; set; }
Create new migration and apply it database (database will have now additional table CombinationObjectTable). After that you start to build a query:
DbSet<CombinationObjectTable> combinationObjectTable = context.Set<CombinationObjectTable>();
StringBuilder wholeQuery = new StringBuilder("DELETE * FROM CombinationObjectTable");
foreach(var obj in myCombinationObject)
{
wholeQuery.Append(string.Format("INSERT INTO CombinationObjectTable(Id1, Id2) VALUES('{0}', '{1}')", obj.Id1, obj.Id2);
}
wholeQuery.Append(
db.Tables
.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.Where( x=>
combinationObjectTable.Any(ct => ct.Id1 == x.Id1 && ct.Id2 == x.Id2)
).ToString();
);
var filteredResults = context.Tables.ExecuteQuery(wholeQuery.ToString());
Thanks to this your main query stays written in linq. If you do not want to add new table to your db this is as well achievable. Add new class CombinationObjectTable to model, generate new migration to add it and afterwards remove code creating that table from migration code. After that apply migration. This way the db schema won't be changed but EF will think that there is CombinationObjectTable in database. Instead of it you will need to create a temporary table to hold data:
StringBuilder wholeQuery = new StringBuilder("CREATE TABLE #TempCombinationObjectTable(Id1 uniqueidentifies, Id2 uniqueidentifier);");
And when you invoke ToString method on your linq query change CombinationObjectTable to #TempCombinationObjectTable:
...
.ToString()
.Replace("CombinationObjectTable", "#TempCombinationObjectTable")
Other thing worth considering would be using query parameters to pass values in INSERT statements instead of just including them in query yourself - this is of course achievable with EF as well. This solution is not fully ready to apply, rather some hint in which direction you may go for the solution.

Can you do something like this:
var result=
db.Tables
.Where(t=>
db.Table3
.Where(x=>x.Prop3 == myFilter)
.Any(a=>a.T1==t.Prop1 || a.T2==t.Prop2)
).ToList();

If you simply want to avoid the intermediate result (and also creating a second intermediary list) you can do the following
var resultsFiltered = db.Tables.Where( x=>
myCombinationObjectId1s.Contains(x.Prop1)
|| myCombinationObjectId2s.Contains(x.Prop2))
.AsEnumerable() // everything past that is done in memory but isn't materialized immediately, keeping the streamed logic of linq
.Where( x=>
myCombinationObject
.Contains(new {Id1 = x.Prop1, Id2 = x.Prop2 })
.ToList();

EF 4: The specified type member '*' is not supported in LINQ to Entities

i have a mapped-class like this:
[Table("MyTable")]
class MyClass
{
//properties; id, name, etc...
private string _queuedToWHTime = string.Empty;
[Column("QueuedToWHTime")]
public string QueuedToWHTime
{
get { return _queuedToWHTime; }
set { _queuedToWHTime = value; }
}
public DateTime? QueuedToWHTime_DateTime
{
get
{
if (!string.IsNullOrWhiteSpace(_queuedToWHTime))
{
return Convert.ToDateTime(_queuedToWHTime);
}
return null;
}
}
}
and a table (MyTable):
CREATE TABLE webnews_in
(
Id INT NOT NULL auto_increment,
QueuedToWHTime VARCHAR (50) NULL
...
PRIMARY KEY (Id)
);
when i trying to query like this:
var searchRslt=(from m in queryableNews
orderby m.QueuedToWHTime_DateTime descending
select m).ToList();
I got a NotSupportedException:The specified type member 'QueuedToWHTime_DateTime' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported.

You cannot query with EF on custom properties. The custom property cannot be translated in SQL.
You can do this to force the orderby to be done 'in-memory'.
var searchRslt = queryableNews
.AsEnumerable()
.OrderBy(m => m.QueuedToWHTime_DateTime)
.ToList();

The issue is here:
var searchRslt=(from m in queryableNews
orderby m.QueuedToWHTime_DateTime descending
select m).ToList();
you are trying to use the property .QueuedToWHTime_DateTime, but that does not exist in the database. You need to use the names that are used in the database. In this case .QueuedToWHTime
So:
var searchRslt=(from m in queryableNews
orderby m.QueuedToWHTime descending
select m).ToList();
If the database propery is not usable in this scenario, you will have to pull the entire list, convert it to an IEnumerable (any will do), then filter/order that IEnumerable by its property.
Something like this:
var result = queryableNews.ToList().OrderbyDescending(x=>x.QueuedToWHTime_DateTime).ToList();

Linq Entity Framework -- can I use a recursive method?

Following on from the excellent answer to my previous question:
Linq Entity Framework generic filter method
I am now trying to understand how I can apply something like recursion to my solution.
To recap, instead of multiple similar declarations of this method:
protected IQueryable<Database.Product> GetActiveProducts( ObjectSet<Database.Product> products ) {
var allowedStates = new string[] { "Active" , "Pending" };
return (
from product in products
where allowedStates.Contains( product.State )
&& product.Hidden == "No"
select product
);
}
I now have a single implementation that accepts an interface type (IHideable) to operate on:
protected IQueryable<TEntity> GetActiveEntities<TEntity>( ObjectSet<TEntity> entities ) where TEntity : class , Database.IHideable {
var allowedStates = new string[] { "Active" , "Pending" };
return (
from entity in entities
where allowedStates.Contains( entity.State )
&& entity.Hidden == "No"
select entity
);
}
This works well and the solution is clean and understandable ( big thanks to https://stackoverflow.com/users/263693/stephen-cleary ).
What I am trying to do now is to apply a similar (or the same?) method to any EntityCollections associated to an EntityObject that also happen to implement IHideable.
I currently make use of GetActiveEntities() like this:
var products = GetActiveEntities( Entities.Products );
return (
from product in products
let latestOrder = product.Orders.FirstOrDefault(
candidateOrder => (
candidateOrder.Date == product.Orders.Max( maxOrder => maxOrder.Date )
)
)
select new Product() {
Id = product.Id ,
Name = product.Name,
LatestOrder = new Order() {
Id = latestOrder.Id ,
Amount = latestOrder.Amount,
Date = latestOrder.Date
}
}
);
In this example I would like to have the Orders EntityCollection also filter by GetActiveEntities(), so that the latest order returned can never be a "hidden" one.
Is it possible to have all EntityCollections implementing IHideable to be filtered - maybe by applying some reflection/recursion inside GetActiveEntities() and calling itself? I say recursion because the best solution would go multiple levels deep walking through the entity graph.
This stuff is stretching my brain!
UPDATE #1 (moved my comments up to here)
Thanks Steve.
Making the method accept IQuerable as suggested gives this error:
'System.Data.Objects.DataClasses.EntityCollection<Database.Order>' does not contain a definition for 'GetActiveEntities' and no extension method 'GetActiveEntities' accepting a first argument of type 'System.Data.Objects.DataClasses.EntityCollection<Database.Order>' could be found (are you missing a using directive or an assembly reference?)
I presume this is because EntityCollection does not implement IQueryable.
I was able to get further by creating a second extension method that explicitly accepted EntityCollection and returned IEnumerable. That compiled but at runtime gave this error:
LINQ to Entities does not recognize the method 'System.Collections.Generic.IEnumerable`1[Database.Order] GetActiveEntities[Order](System.Data.Objects.DataClasses.EntityCollection`1[Database.Order])' method, and this method cannot be translated into a store expression.
I also tried calling AsQueryable() on the EntityCollection and returning IQueryable but the same error came back.

Edited to use ObjectSet instead of EntityCollection
I recommend using my solution, but as an extension method, which can be applied to any query:
public static IQueryable<TEntity> WhereActive<TEntity>(
this IQueryable<TEntity> entities)
where TEntity : class , Database.IHideable
{
var allowedStates = new string[] { "Active", "Pending" };
return (
from entity in entities
where allowedStates.Contains(entity.State)
&& entity.Hidden == "No"
select entity
);
}
This can then be used as such:
var products = Entities.Products.WhereActive();
return (
from product in products
let latestOrder = (
from order in Entities.Orders.WhereActive()
where order.ProductId == product.Id
orderby candidateOrder.Date descending
select order).FirstOrDefault()
select new Product()
{
Id = product.Id,
Name = product.Name,
LatestOrder = new Order()
{
Id = latestOrder.Id,
Amount = latestOrder.Amount,
Date = latestOrder.Date
}
}
);

LINQ to SQL: Advanced queries against lists, arrays and lists of objects

Single and multiple lists
Consider the following lists:
List<Int32> appleIdentities = new List<int>(new[] { 1, 2, 3 });
List<Int32> chocolateIdentities = new List<int>(new[] { 2, 3, 4 });
List<Int32> icecreamIdentities = new List<int>(new[] { 11, 14, 15, 16 });
Using LINQ to SQL; is it possible to wite a statement which translates into:
SELECT
DesertsID,
DesertsName
FROM
Deserts
WHERE
Deserts.AppleIdentity IN (1, 2, 3) AND
Deserts.ChocolateIdentity IN (2, 3, 4) AND
Deserts.IcecreamIdentity IN (11, 14, 15m 16)
If yes; how would the code look if I wanted to query my database of deserts against just the appleIdentities list?
Arrays
Consider the following arrays:
Int32[] appleIdentities = new[] {1, 2, 3, 4};
String[] chocolateNames = new[] {"Light", "Dark"};
Using LINQ to SQL; is it possible to wite a statement which translates into:
SELECT
DesertsID,
DesertsName
FROM
Deserts
WHERE
Deserts.AppleIdentity IN (1, 2, 3) AND
Deserts.ChocolateName IN ('Light', 'Dark')
If yes; how would the code look if I wanted to query my database of deserts against just the appleIdentities array?
List of objects
Consider the following:
public class Identities
{
public Int32 appleIdentity { get; set; }
public String chokolateName { get; set; }
}
List<Identities> identities = new List<Identities>(new[] {
new Identities { appleIdentity = 1, chokolateName = "Light" },
new Identities { appleIdentity = 2, chokolateName = "Dark" },
});
Using LINQ to SQL; is it possible to wite a statement which translates into:
SELECT
DesertsID,
DesertsName
FROM
Deserts
WHERE
Deserts.AppleIdentity IN (1, 2) AND
Deserts.ChocolateName IN ('Light', 'Dark')
If yes; how would the code look if I wanted to query my database of deserts against just the appleIdentity-property on my list of Identities objects?
This is branch off of LINQ to SQL query against a list of entities

how would the code look if I wanted to
query my database of deserts against
just the appleIdentities list?
You can compose a linq query in multiple statements, like so, and select at runtime which filters your want to use in your where clause.
var query = db.Desserts;
if (filterbyAppleIdentity)
query = query.Where( q => appleIdentities.Contains(q.DesertsID));
if (filterbyChocolateIdentities)
query = query.Where( q => chocolateIdentities.Contains(q.DesertsID));
if (filterbicecreamIdentities)
query = query.Where( q => icecreamIdentities.Contains(q.DesertsID));
var deserts = query.ToList();
you can also write an extension method to do this without if statements: (Edit fixed typo, return type should be IQueriable
public static class LinqExtensions {
public IQueriable<T> CondWhere<T>(this IQueriable<T> query, bool condition, Expression<Func<T,bool>> predicate) {
if (condition)
return query.Where(predicate);
else
return query;
}
}
and write your linq query like this:
var deserts = db.Desserts;
.CondWhere(filterbyAppleIdentity, q => appleIdentities.Contains(q.DesertsID));
.CondWhere(filterbyChocolateIdentities, q => chocolateIdentities.Contains(q.DesertsID));
.CondWhere(filterbicecreamIdentities, q => icecreamIdentities.Contains(q.DesertsID)).ToList();
Another way to do it is to union the id lists:
var deserts = db.Deserts
.Where( d => appleIdentities.Union(chocolateIdentities).Union(icecreamIdentities).Contains(d.DesertsID);
For a list of objects you can use .Select extension method to project your list into a int or string IEnumerable and you can use contains in the query in the same way:
var deserts = db.Deserts
.Where(d =>
identities.Select(i => i.appleIdentity).Contains(d => d.DesertID) &&
identities.Select(i => i.chokolateName).Contains(d => d.DesertsName)
)

Sure - just use Contains - using Northwind as an example:
var qry = from cust in ctx.Customers
where custIds.Contains(cust.CustomerID)
&& regions.Contains(cust.Region)
select cust; // or your custom projection

Well, you can try:
var query = from dessert in db.Desserts
where appleIdentities.Contains(dessert.AppleIdentity)
&& chocolateIdentities.Contains(dessert.ChocolateIdentity)
&& iceCreamIdentities.Contains(dessert.IceCreamIdentity)
select new { dessert.Id, dessert.Name };
I believe that's okay, although it'll fail when the lists get big enough IIRC. That should be okay for lists and arrays.
I'm not sure about your third query though - I think you'd need a list for each of the separate Contains calls.

As others have said, LinqToSql will translate Contains to IN.
There's some caveats:
this translation works for List<T>.Contains(), but doesn't work for IList<T>.Contains(). Does it work for arrays? I don't know.
This translation will happily translate as many elements as you like - each element becomes a sql parameter. SQL Server 2008 has an approx 2000 parameter limit and will throw sql exceptions at you if you try this with a collection that is too big.
This translation, when applied to a collection of strings, will produce nvarchar parameters. This could be a serious problem if the target column is varchar and you want to use the index on this column. Sql Server will convert the index, instead of the parameters... which involves reading and converting every string in the whole index.
Here's some code for your List of Objects question:
List<int> someIDs = identities
.Select(x => x.appleIdentity).ToList();
List<string> someStrings = identities
.Select(x => x.chokolateName).ToList();
var query = db.Desserts.Where(d =>
someIDs.Contains(d.AppleIdentity) &&
someStrings.Contains(d.ChocolateName)
)