We Keep Coding

EFCore join values

I'm trying to join a big table to a small list of data pairs using EFCore 2.1.1. I want this join to happen server-side, rather than trying to download the whole table, e.g translating to something like:
SELECT a.*
FROM Groups AS a
INNER JOIN (VALUES (1, 'admins'), (2, 'support'), (1, 'admins')) AS b(organization_id, name)
ON a.organization_id = b.organization_id AND a.name = b.name;
or something equivalent (e.g. using common table expressions). Is this possible? If so, how? Passing a list of objects to a LINQ .join seems to always get handled client-side.
Due to massive testing debt and the EFCore 3 breaking change on client-side evaluation, upgrading is not an option for us at this time (but answers relevant to newer versions may help us push management)

If you expect that EF Core 3.x can support this, you are wrong. If you plan to upgrade your application, better think about EF Core 6 and .net 6.
Anyway I know several options:
With extension method FilterByItems or similar
var items = ...
var query = context.Groups
.FilterByItems(items, (q, b) => q.organization_id == b.organization_id && q.name == i.name, true);
With third party extension inq2db.EntityFrameworkCore version 2.x, note that I'm one of the creators. It will generate exactly the same SQL as in question.
var items = ...
var query =
from g in context.Groups
join b in items on new { g.organization_id, g.name } equals new { b.organization_id, b.name }
select g;
var result = query.ToLinqToDB().ToList();

You could solve this problem with a Table Value Parameter.
First define a database type;
IF TYPE_ID(N'[IdName]') IS NULL
CREATE TYPE [IdName] AS TABLE (
[Id] int NOT NULL
[Name] nvarchar(max) NOT NULL
)
Then you can build an SqlParameter from an IEnumerable;
public class IdName
{
public int Id { get; set; }
[Required]
public string Name { get; set; }
public static SqlParameter ToParameter(IEnumerable<IdName> values)
{
var meta = new SqlMetaData[]{
new SqlMetaData(nameof(Id), SqlDbType.Int),
new SqlMetaData(nameof(Name), SqlDbType.NVarChar, int.MaxValue)
};
return new SqlParameter()
{
TypeName = nameof(IdName),
SqlDbType = SqlDbType.Structured,
Value = values.Select(v => {
var record = new SqlDataRecord(meta);
record.SetInt32(0, v.Id);
record.SetString(1, v.Name);
return record;
})
};
}
}
Then define an EF Core query type, and you can turn that SqlParameter into an IQueryable;
public void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<IdName>(e =>
{
e.HasNoKey();
e.ToView(nameof(IdName));
});
}
public static IQueryable<IdName> ToQueryable(DbContext db, IEnumerable<IdName> values)
=> db.Set<IdName>().FromSqlInterpolated($"select * from {ToParameter(values)}");
And now you can use that IQueryable in a Linq join.

EF Core 5.0 Union Linq Query with sub selects not working

Goal:
I want to combine information from two tables (entityA & entityB) with different Properties to one unionDto. I am trying to implement this with a union operation to filter different entities in the Database at the same time.
But the structure I am using requires Versions which need to be filtered before the union query.
Some Additional Information:
So what I am trying to do in the last query is: First I project information from table "entityA" and "entityB" to a universal "anonymous" type (with union). Then I am trying to apply pagination and then I am trying to "project" the new anonymous result to a UnionDto. So this would result in one UnionDto which has "information from 2 different tables".
I have created an example of my problem with two entities which have Versions:
Entities:
class entityA {
public List<VersionA> Versions;
public Guid CreatedBy;
}
class entityB {
public List<VersionB> Versions;
public Guid CreatedBy;
}
class VersionA {
public string TitleA;
public Instant EffectiveTo;
}
class VersionB {
public string TitleB;
public Instant EffectiveTo;
}
class UnionDto{
public string Title;
public Guid Creator;
}
I am setting up the query like this:
var queryA = databaseContext.Set<entityA>()
.Select(entity => new
{
Versions = entity.Versions
.Where(version => version.EffectiveTo > now) /* Filtering newest entity Version */
.Select(versionDetail => new /* Selecting only the Title of this Version */
{
Title = versionDetail.TitleA
})
.ToList(),
Creator = entity.CreatedBy,
});
var queryB = databaseContext.Set<entityB>()
.Select(entity => new
{
Versions = entity.Versions
.Where(version => version.EffectiveTo > now)
.Select(versionDetail => new
{
Title = versionDetail.TitleB
})
.ToList(),
Creator = entity.CreatedBy,
});
Executing the query:
var unionDto = await queryA
.Union(queryB)
.Skip(0)
.Take(20)
.Select(x => new UnionDto
{
Title= x.Versions.FirstOrDefault() == null ? null :
x.Versions.FirstOrDefault().Title,
Creator= x.Creator,
})
.ToListAsync();
It seems like that i am not able to use sub selects inside a union query and I am getting the following error:
Set operations: support when placed after client evaluation in
projection #16243
I dont know, what I have to do to get around this issue, since I dont really want to go throught all of my entities seperatly with seperate Database queries.
Currently using Ef core Version: 5.0.100-preview
Now this is just an example. This would be required for at least 10 entities, which would result in high Database traffic if done for each entity seperatly.
Any ideas?

If you need only title of first version from each recordset, your query can be simplified and EF Core can translate this query.
var queryA =
from entity in databaseContext.Set<entityA>()
from version in entity.Versions
where version.EffectiveTo > now
select new
{
Title = version.Title,
Creator = entity.CreatedBy
}
var queryB =
from entity in databaseContext.Set<entityB>()
from version in entity.Versions
where version.EffectiveTo > now
select new
{
Title = version.Title,
Creator = entity.CreatedBy
}
var versions = queryA.Union(queryB);
var unionDto = await versions
.Skip(0)
.Take(20)
.Select(x => new UnionDto
{
Title = x.Title,
Creator = x.Creator,
})
.ToListAsync();

Complex join on multiple fields

I am trying to rebuild a SQL query with Linq, but run into a combined join statement. For simplicity lets limit this two two entitities A and B. The SQL part is as follows:
FROM Entity A
INNER JOIN Entity B WITH(NOLOCK) ON ISNULL(B.property1, B.property2) = A.property1 AND B.property3 = a.property2
How can I implement this using Linq?
I currently have the following:
from quotation in _context.Quotation
join proj in _context.Planning on new { quotation.PlanningKey, quotation.BudgetKey } equals new { proj.PlanningKey, proj.BudgetKey }
join act in _context.Planning on new { proj.ProjectKey, proj.BudgetKey } equals new { act.ProjectKey, act.BudgetKey}
where ...
orderby ...
select new
{
...
};
However this is not taking into account the "Or" part. I want to compare property 1 or 2 from entity B to property 1 from entity A AND property 3 from entity B to property 2 from entity A.

The linq Join method only supports equality predicates, and AND'ing together multiple equality predicates by using anonymous objects. If you need anything else than that, you need to express it using Where instead.

You can try following snippet
enityA.Join(
enityB,
entA => new { Property1 = entA.property1Post_id, Property2 = entA.property2Post_id } ,
entB => new { Property1 = entB.property1Post_id?? entB.property2Meta, Property2 = entB.property3Meta } ,
(entA, entB) => new { EntityA = entA, EntityB = entB }
);

entity framwork core generates weird sql

I have an existing LINQ query that I am trying to optimize. I have the following entity Types (simplified)
public class Account
{
public int Id { get; set; }
public IEnumerable<OpportunityInfo> Opportunities { get; set; }
}
public class Opportunity
{
public int Id { get; set; }
public string Name { get; set; }
public bool Active { get; set; }
public IEnumerable<Quote> Quotes { get; set; }
}
public class Quote
{
}
It is a standard hierarchy of Account to Opportunity to Quote. Nothing Special. I have the following query that I am using on an ASP.NET Core controller index method. I am starting from Quote and working backwards because there is dynamic query logic between the query and opportunityQuotes that must be Quote based. Otherwise I would start from the top direction.
var query = from o in Quotes select o;
additional query logic (filtering and sorting)
var opportunityQuotes = from o in query
group o by new
{
accountId = o.Opportunity.AccountId,
accountName = o.Opportunity.Account.Name,
active = o.Opportunity.Account.Active,
}
into p
select new
{
Id = p.Key.accountId,
Name = p.Key.accountName,
Active = p.Key.active,
Opportunities =
(from q in p
group q by new
{
Id = q.Opportunity.Id,
Name = q.Opportunity.Name,
Active = q.Opportunity.Active
}
into r
select new
{
Name = r.Key.Name,
Id = r.Key.Id,
Active = r.Key.Active,
Quotes = r
})
};
opportunityQuotes.Dump();
This query generates the following SQL.
SELECT [o].[Id], [o].[ARRValue], [o].[AccountId], [o].[AdjustedArr], ...
FROM [Quotes] AS [o]
LEFT JOIN [Opportunities] AS [o.Opportunity] ON [o].[OpportunityId] = [o.Opportunity].[Id]
INNER JOIN [Accounts] AS [o.Account] ON [o].[AccountId] = [o.Account].[Id]
ORDER BY [o].[AccountId], [o.Account].[Name], [o.Account].[Active]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
SELECT [q.Opportunity0].[Id], [q.Opportunity0].[Name], [q.Opportunity0].[Active]
FROM [Opportunities] AS [q.Opportunity0]
GO
In reality it generates on query for each opportunity, but I left that out for brevity sake. In my opinion EF should not generate a separate query for each quote. In fact if I comment out the .Name and .Active key parameters in the query as shown below:
group q by new
{
Id = q.Opportunity.Id,
// Name = q.Opportunity.Name,
// Active = q.Opportunity.Active
}
and comment out the correspond variables in the select clause it generates much cleaner sql.
SELECT [o].[Id], [o].[ARRValue], [o].[AccountId], ...
FROM [Quotes] AS [o]
LEFT JOIN [Opportunities] AS [o.Opportunity] ON [o].[OpportunityId] = [o.Opportunity].[Id]
INNER JOIN [Accounts] AS [o.Account] ON [o].[AccountId] = [o.Account].[Id]
ORDER BY [o].[AccountId], [o.Account].[Name], [o.Account].[Active]
GO
The reason I am confused is that .Name and .Active are in the exact same object, they are grouped in the key in the same way as the .Id field, and therefore I don't see why EF would change its behavior just by adding additional group values. Can someone explain the behavior?

Let's take a step back and look at it from a different perspective: If you were to write the SQL query manually, and wanted to fetch all the data required in one query, you would get a lot of duplicate data for the opportunities and account. You could also do this here:
var query = from o in Quotes select o;
var oppQuotes = from o in query
select new
{
AccountId = o.Opportunity.Account.Id,
AccountName = o.Opportunity.Account.Name,
// ... and so on, with all the fields you expect to use.
OpportunityId = o.Opportunity.Id,
OpportunityName = o.Opportunity.Name,
// ... and so on, with all the fields you expect to use.
QuoteId = o.Id,
QuoteName = o.Name,
// ... and again, you get the point.
};
Then, just do an .AsEnumerable() on it, and perform the grouping in your C# code. The database won't be able to optimize anything anyways.
var opportunityQuotes = from q in oppQuotes.AsEnumerable()
group q by new { q.AccountId, q.AccountName }
into accounts
// ... and so on.
For your question, why EF is creating the strange query, I'm at a loss.
In any case, it is always good to be thinking about how YOU would create the sql code to get the data you want most efficiently and not rely on EF to "do the right thing". In many cases it will, in others it will completely blow up in your face. When you want a query, think of the SQL and then translate that to EF code. If you tell it specifically, what you want, you will get it.

How to filter columns on child table in a one to many relationship

I am trying to write a query that filters fields in a parent table, includes a child table and also filters fields in the child table.
I have sort of got this working by putting ToList in the child selector but this just feels wrong to me. Is this the right way to do this?
Example:
var query = _context.Set<order_header>()
.Where(oh => oh.customer == accountNo)
.Include(oh => oh.route_details)
.Select(oh => new order_header()
{
customer = oh.customer,
order_no = oh.order_no,
//other columns omitted
route_details = oh.route_details
.Select(rd => new route_detail() { route_code = rd.route_code})
.ToList()//this is odd
});
return query.ToList();
Edit:
I've enabled a SQL trace and I can see this is doing a separate query to get the children for every parent row. So this is definitely the wrong way to do things.
I'm starting to think I'll have to select the results into an anonymous type and generate the EF models afterwards.
Edit2:
I have now removed the ToList in the sub-query select but SQL trace shows this as still running a query for every parent row.
Code:
var query = _context.Set<order_header>()
.Where(oh => oh.customer == accountNo)
.Include(oh => oh.route_details)
.Select(oh => new
{
customer = oh.customer,
order_no = oh.order_no,
//other columns omitted
route_details = oh.route_details.Select(rd => rd.route_code)
});
var result = query.ToList();
var list = new List<order_header>();
list.AddRange(result.Select(a =>
new order_header()
{
customer = a.customer,
order_no = a.order_no,
//other columns omitted
route_details = a.route_details.Select(rc => new route_detail() { route_code = rc }).ToList()
}));
return list;
Edit3
As requested, the SQL trace:
Parent query
exec sp_executesql N'SELECT [oph].[customer], [oph].[order_no], [oph].[customer_order_no], [oph].[date_received], [oph].[date_required], [oph].[date_despatched], [oph].[status], [oph].[to_reference], [oph].[from_reference], [oph].[nett_value]
FROM [scheme].[order_header] AS [oph]
WHERE [oph].[customer] = #__accountNo_0',N'#__accountNo_0 varchar(8)',#__accountNo_0='ACC_NO'
Child queries
exec sp_executesql N'SELECT [avl].[route_code]
FROM [scheme].[route_detail] AS [avl]
WHERE #_outer_order_no = [avl].[ldordno]',N'#_outer_order_no varchar(10)',#_outer_order_no='1A469499 '
imgur link
http://i.imgur.com/Q4ATQiU.png
note that the schema names are different in the image as I have been editing them for the question.

You can do a select with anonymous types which should then result in one query with appropriate joins, something like this (not sure I got your navigation properties right):
var query = from oh in _context.Set<order_header>()
where oh.customer == accountNo
select new
{
oh,
oh.route_details,
oh.customer,
// other navigation properties to include
route_details = from rd in oh.route_details
// your child table filtering here
select new
{
rd,
rd.route_code,
// other child nav properties to include
}
};
return query.AsEnumerable().Select(m => m.oh).ToList();