I have 3 tables: Person, PersonFriend, PersonGroup.
Using LINQ, i want to join the 3 tables, filter using a dynamically generated where clause, and select custom columns with flattened rows (flattened one-to-many relationship table columns).
Pseudo-SQL design:
CREATE TABLE Person (int id, varchar socialclass, date createddate);
CREATE TABLE Person_Friend (int id, id personid references person.id, id friendpersonid references person.id, varchar friendtype);
CREATE TABLE Person_Group (int id, int memberid references person.id, varchar membershiplevel);
Entities:
public class Person
{
public int Id { get; set; }
public string SocialClass { get; set; }
public DateTime? CreatedDate { get; set; }
public ICollection<PersonFriend> Friend { get; set; }
public ICollection<PersonGroup> Group { get; set; }
}
public class PersonFriend
{
public int Id { get; set; }
public int PersonId { get; set; }
public int FriendPersonId { get; set; }
public string FriendType { get; set; }
}
public class PersonGroup
{
public int Id { get; set; }
public int MemberId { get; set; }
public string MembershipLevel { get; set; }
}
query syntax LINQ:
var queryResult = from person in _context.Person
join friend in _context.PersonFriend on person.Id equals friend.FriendPersonId
join group in _context.PersonGroup on person.Id equals group.MemberId
where (friend.PersonId == 1 && friend.FriendType == "type1") || (friend.PersonId == 3 && friend.FriendType == "type2") || ...
select new { person.Id, person.SocialClass, person.CreatedDate, friend.FriendPersonId, friend.FriendType, group.Id, group.MembershipLevel };
Notice the where clause; Given a list of { PersonId, FriendType } object, I want to build the where clause like above.
Since I could not figure building a dynamic where clause for a query syntax LINQ,
I tried converting it to the Method syntax LINQ statement so i can leverage the PredicateBuilder (http://www.albahari.com/nutshell/predicatebuilder.aspx) but I run into the problem during Selecting one-to-many things into a flattened object.
var methodResult = _context.Person
.Include(x => x.Friend)
.Include(x => x.Group)
.Select(person => new { person.Id, person.SocialClass, person.CreatedDate, person.friend.FriendPersonId, person.friend.FriendType, person.group.Id, person.group.MembershipLevel });
notice that the above Select is not possible because friend is a ICollection.
I also tried using the above query syntax LINQ statement without the where clause, making it return a object instead of an annonymous object, and then calling the method .Where() with the predicate builder. But the built expression runs into LINQ => Entity Framework SQL conversion error and executes the where in the application, not in DB.
var queryResultWithoutWhere = from person in _context.Person
join friend in _context.PersonFriend on person.Id equals friend.FriendPersonId
join group in _context.PersonGroup on person.Id equals group.MemberId
select new SelectedObject { PersonId = person.Id, SocialClass = person.SocialClass, CreatedDate = person.CreatedDate, FriendId = friend.FriendPersonId, FriendType = friend.FriendType, GroupId = group.Id, MembershipLevel = group.MembershipLevel };
var predicate = PredicateBuilder.New<SelectedObject>(false);
foreach (var searchObject in searchRequestObjects)
{
predicate.Or(p => p.FriendPersonId == searchObject.FriendPersonId && p.FriendType == searchObject.FriendType);
}
var result = queryResultWithoutWhere.Where(predicate).ToList();
I feel like I tried everything I could, and I cannot seem to generate this SQL. Last resort would be writing a raw SQL string and then executing it, but I really would like to get this working with Entity Framework.
How would I accomplish creating a dynamic where clause, select into a custom flattened object, and have entity framework generate the SQL?
You can use SelectMany to flatten the collections:
var methodResult = Persons
.Include(x => x.Friend)
.Include(x => x.Group)
.SelectMany(person =>
person.Friend.SelectMany(friend =>
person.Group.Select(group =>
new {
person.Id,
person.SocialClass,
person.CreatedDate,
friend.FriendPersonId,
friend.FriendType,
GroupId = group.Id,
group.MembershipLevel
}
)
)
);
Related
So the equivalent of this query:
select * from car
left join parts ON car.Id = parts.carId
where parts.MemberId = 1
is this, in EntityFrameworkCore LINQ , using an IQueryable which has already selected car.Include(x => x.parts):
queryable = queryable.Where(x =>
x.parts.Select(y => y.MemberId).Contains(1);
But how can I convert the following SQL to LINQ, so that it includes rows from the left car table that have no respective MemberId entries in the parts table?
select * from car
left join parts ON car.Id = parts.CarId and parts.MemberId = 1
Models:
public class Car
{
public int Id { get; set; }
public virtual ICollection<Part> Parts { get; set; }
}
public class Parts
{
public int Id { get; set; }
public int CarId { get; set; }
public virtual Car { get; set; }
public int MemberId { get; set; }
}
A filtered Include does exactly what you want:
var cars = context.Cars
.Include(c => c.Parts.Where(p => p.MemberId == 1));
This doesn't generate the shorter join statement with a composite condition, but an outer join to a filtered subquery on Parts, to the same effect.
queryable = queryable
.Where(x => x.parts.Select(y => y.MemberId).Contains(1) || !x.parts.Any());
Try it like that:
queryable = queryable.Include(x => x.parts).Where(x =>
x.parts.Any(y => y.MemberId == 1).ToList();
How can I get the list of products for specific project without including project? Something similar to
select p.Id, p.Name
from product p
inner join ProjectProduct j on p.Id = j.ProductId and j.ProjectId = #Id
The closest solution that I have is
var products = await (from proj in context.Project select proj)
.Where(proj => proj.Id == 1)
.Select(proj => proj.Product)
.ToListAsync();
but this returns a List<List<Product>>
in addition I need to be able to add the order by and pagination to that
I know that I can do it by raw sql like this
var param1 = new SqlParameter("#Id", id);
var query = #"select * from product p
inner join ProjectProduct j
on p.Id = j.ProductId and j.ProjectId = #Id ";
var queryable = context.product.FromSqlRaw(query, param1).AsQueryable();
await HttpContext.AddPaginationToHeader(queryable);
var products = await queryable.OrderBy(x => x.Name).Paginate(paginationDTO).ToListAsync();
But I was hoping to do it by linq to entity
public class Project
{
public int Id { get; set; }
public string Name { get; set; }
public List<Product> Product { get; set; }
}
public class Product
{
public int Id { get; set; }
public string Name { get; set; }
public List<Project> Project { get; set; }
}
public class ProjectProduct
{
public int ProjectId { get; set; }
public int ProductId { get; set; }
}
Thanks to Harald Coppoolse's response on another post
and for those that have the same problem
here are the steps
1- we define our query as queryable
var queryable = context.Project
.Where(proj => proj.Id == id)
.SelectMany(p => p.Product).AsQueryable();
2- Now we can pass the query to SQL server like this
var products = await queryable.OrderBy(x =>
x.Name).Skip(...).Take(...).ToListAsync();
for Skip and Take we can create a generic Extension for IQueryable
and add a method to that with return of
return queryable
.Skip((paginationDTO.Page - 1) * paginationDTO.PageSize)
.Take(paginationDTO.PageSize);
and of course our paginationDTO is nothing more than a POCO class with 2 properties of page and pageSize
try this:
var products = await context.Project
.Where(proj => proj.Id == 1)
.Select( proj=>proj.Products)
.FirstOrDefaultAsync();
if you want to order product after this you can do this common way
products= products.OrderBy(p=>p.Name).ToList();
it can be put in one line but EF sometimes generates a very ridiculous Sql script, so better to do this separately.
Hi good day I am new with Entity Framework. I just wanna to know if there is a way I could improve my implementation. Here are the codes.
public async Task<List<Record>> GetRecordsByBatchId(string batchId, string source)
{
List<string> idList = new List<string>();
//[1] Get all parent ID from table 1 with a filter of source and batchId
var parentIds= await _context.Set<FirstTable>()
.Where(a => a.IsActive
&& a.BatchId.Equals(batchId)
&& a.Source.Equals(source)).Select(b => b.ParentId).ToListAsync();
if (parentIds.Count() == 0)
{
return new List<Record>();
}
//[2] Query idNumber of each parentId from [1] to SecondTable
List<long> idNumber = await _context.Set<SecondTable>()
.Where(a => parentIds.Contains(a.Id))
.Select(b => b.IdNumber).ToListAsync();
//[3] Query Record/s that contains idNumber from previous query [2]. it is possible that 1 or
//more records has same idNumber
List<Risk> recordByIdNumber = await _context.Set<SecondTable>()
.Where(a => idNumber.Contains(a.IdNumber)).ToListAsync();
//[4] In this part I just want to group the records in [3] by Id number and sort each group
//by its endorsementNumber in descending order and return the record with highest endorsement
//number for each group
return (from record in recordByIdNumber
group record by record.IdNumber into g
orderby g.Key
select g.OrderByDescending(risk =>risk.EndorsementNumber).FirstOrDefault()).ToList();
}
}
The model for the FirstTable
public class FirstTable
{
public Guid? ParentId{ get; set; }
public string BatchId { get; set; }
public string Source { get; set; }
public bool IsActive { get; set; }
}
The model for the SecondTable
public class SecondTable
{
public Guid Id{ get; set; }
public int EndorsementNumber { get; set; }
public long IdNumber { get; set; }
}
Note: I just include the necessary properties in the model.
This approach is working as expected. I just wanna know if there is a possibility that these queries could be optimized that there is only 1 query for the SecondTable table.
Any help would be greatly appreciated, thanks in advance.
Yes, queries 1-3 can and should be combined. In order to do that you need, to have navigation properties in your model. It seems that there is one-to-many relationship between FirstTable and SecondTable. Let's use Customer and Order instead.
class Customer {
int CustomerId
string BatchId
ICollection<Order> Orders
}
class Order {
int OrderId
int CustomerId
Customer Customer
Risk Risk
}
in which case you just write third query as
List<Risk> = await _context.Orders.Where(o => o.Customer.BatchId == batchId)
.Select(o => o.Risk).ToListAsync();
Obviously, I am only guessing the structure and the relationship. But hopefully, this can get you started. For me Contains() is "code smell". There is a high chance that there will be large list out of your first query, and contains() will produce a huge IN clause in the database, that can easily crash the system
var parentIds = _context.Set<FirstTable>()
.Where(a => a.IsActive
&& a.BatchId.Equals(batchId)
&& a.Source.Equals(source)).Select(b => new { b.parentId });
var risks = await (from s in _context.Set<SecondTable>()
join p in parentIds on s.Id equals p.parentId
join r in _context.Set<SecondTable>() on s.IdNumber equals r.IdNumber
select r).GroupBy(r=>r.IdNumber)
.Select(r=> r.OrderByDescending(risk =>risk.EndorsementNumber).FirstOrDefault())
.ToArrayAsync();
return risks;
You can have 1 query instead of 3. It will perform better as the number of the rows from the first query grows.
EDIT: As #SvyatoslavDanyliv mentioned in the comments, group-take operations may not work depending on the version of the EF and the provider you use. You may need to separate the query and the group by operation like below :
var result = await (from s in _context.Set<SecondTable>()
join p in parentIds on s.Id equals p.parentId
join r in _context.Set<SecondTable>() on s.IdNumber equals r.IdNumber
select r).ToArrayAsync();
var risks = result.GroupBy(r=>r.IdNumber)
.Select(r=> r.OrderByDescending(
risk =>risk.EndorsementNumber).FirstOrDefault())
.ToArray();
return risks;
I have SQL query like this
SELECT T.*
FROM
(
SELECT ServiceRecords.DistrictId, Districts.Name as DistrictName, COUNT(Distinct(NsepServiceRecords.ClientRegNo)) AS ClientsServedCount
FROM ServiceRecords
INNER JOIN Districts ON ServiceRecords.DistrictId = Districts.ID
INNER JOIN NsepServiceRecords ON NsepServiceRecords.ServiceRecordId = ServiceRecords.Id
WHERE ServiceRecords.CreatedAtUtc >= #StartDate
AND ServiceRecords.CreatedAtUtc <= #EndDate
AND ServiceRecords.DistrictId = #DistrictId
GROUP BY ServiceRecords.DistrictId, Districts.Name
) AS T
ORDER BY T.DistrictName ASC, T.DistrictId
Query results:
DistrictId DistrictName ClientsServedCount
8d059005-1e6b-44ad-bc2c-0b3264fb4567 Bahawalpur 117
27ab6e24-50a6-4722-8115-dc31cd3127fa Gujrat 492
14b648f3-4912-450e-81f9-bf630a3dfc72 Jhelum 214
8c602b99-3308-45b5-808b-3375d61fdca0 Lodhran 23
059ffbea-7787-43e8-bd97-cab7cb77f6f6 Muzafarghar 22
580ee42b-3516-4546-841c-0bd8cef04df9 Peshawar 211
I'm struggling converting this to LINQ to entities query. I want to get same results (except District Id column) using LINQ.
I have tried like this, but not working as expected. Can somebody tell me what I'm doing wrong?
_dbContext.ServiceRecords
.Include(x => x.District)
.Include(x=>x.NsepServiceRecords)
.GroupBy(x => x.DistrictId)
.Select(x => new DistrictClientsLookUpModel
{
DistrictName = x.Select(record => record.District.Name).FirstOrDefault(),
ClientsServedCount = x.Sum(t=> t.NsepServiceRecords.Count)
});
Model classes are like this
public class BaseEntity
{
public Guid Id { get; set; }
}
public class NsepServiceRecord : BaseEntity
{
public DateTime CreatedAtUtc { get; set; }
public Guid ServiceRecordId { get; set; }
public string ClientRegNo { get; set; }
// other prop .......
public virtual ServiceRecord ServiceRecord { get; set; }
}
public class ServiceRecord : BaseEntity
{
public DateTime CreatedAtUtc { get; set; }
public string DistrictId { get; set; }
public virtual District District { get; set; }
public virtual ICollection<NsepServiceRecord> NsepServiceRecords { get; set; }
}
public class DistrictClientsLookUpModel
{
public string DistrictName { get; set; }
public int ClientsServedCount { get; set; }
}
I'm using Microsoft.EntityFrameworkCore, Version 2.2.4
EDIT
I have also tried like this
var startUniversalTime = DateTime.SpecifyKind(request.StartDate, DateTimeKind.Utc);
var endUniversalTime = DateTime.SpecifyKind(request.EndDate, DateTimeKind.Utc);
return _dbContext.NsepServiceRecords
.Join(_dbContext.ServiceRecords, s => s.ServiceRecordId,
r => r.Id, (s, r) => r)
.Include(i => i.District)
.Where(x => x.DistrictId == request.DistrictId
&& x.CreatedAtUtc.Date >= startUniversalTime
&& x.CreatedAtUtc.Date <= endUniversalTime)
.OrderBy(x => x.DistrictId)
.GroupBy(result => result.DistrictId)
.Select(r => new DistrictClientsLookUpModel
{
DistrictName = r.Select(x=>x.District.Name).FirstOrDefault(),
ClientsServedCount = r.Sum(x=>x.NsepServiceRecords.Count())
});
Another try,
from s in _dbContext.ServiceRecords
join record in _dbContext.NsepServiceRecords on s.Id equals record.ServiceRecordId
join district in _dbContext.Districts on s.DistrictId equals district.Id
group s by new
{
s.DistrictId,
s.District.Name
}
into grp
select new DistrictClientsLookUpModel
{
DistrictName = grp.Key.Name,
ClientsServedCount = grp.Sum(x => x.NsepServiceRecords.Count)
};
It takes too long, I waited for two minutes before I killed the request.
UPDATE
EF core have issues translating GroupBy queries to server side
Assuming the District has a collection navigation property to ServiceRecord as it should, e.g. something like
public virtual ICollection<ServiceRecord> ServiceRecords { get; set; }
you can avoid the GroupBy by simply starting the query from District and use simple projection Select following the navigations:
var query = _dbContext.Districts
.Select(d => new DistrictClientsLookUpModel
{
DistrictName = d.Name,
ClientsServedCount = d.ServiceRecords
.Where(s => s.CreatedAtUtc >= startUniversalTime && s.CreatedAtUtc <= endUniversalTime)
.SelectMany(s => s.NsepServiceRecords)
.Select(r => r.ClientRegNo).Distinct().Count()
});
You don't appear to be doing a join properly.
Have a look at this:
Join/Where with LINQ and Lambda
Here is a start on the linq query, I'm not sure if this will give you quite what you want, but its a good start.
Basically within the .Join method you need to first supply the entity that will be joined. Then you need to decide on what they will be joined on, in this case district=> district.Id, serviceRecord=> serviceRecord.Id.
_dbContext.ServiceRecords
.Join( _dbContext.District,district=> district.Id, serviceRecord=> serviceRecord.Id)
.Join(_dbContext.NsepServiceRecords, Nsep=> Nsep.ServiceRecord.Id,district=>district.Id)
.GroupBy(x => x.DistrictId)
.Select(x => new DistrictClientsLookUpModel
{
DistrictName = x.Select(record => record.District.Name).FirstOrDefault(),
ClientsServedCount = x.Sum(t=> t.NsepServiceRecords.Count)
});
Consider i have below entities:
public class Root
{
public long Id { get; set; }
}
public class School : Root
{
public long StudentId { get; set; }
public Student Student { get; set; }
public Teacher Teacher { get; set; }
public long TeacherId { get; set; }
}
public class Student : Root
{
}
public class Teacher : Root
{
}
Now, after this fix in EF i can build left join query like this:
ctx.Schools
.GroupJoin(ctx.Teachers, school => school.TeacherId, teacher => teacher.Id,
(school, teachers) => new { school, teachers })
.SelectMany(info => info.teachers.DefaultIfEmpty(),
(info, teacher) => new { info.school, teacher })
.Where(info => info.school.Id == someSchoolId)
.Select(r => r.school);
or like this:
from school in ctx.Schools
join teacher in ctx.Teachers on school.TeacherId equals teacher.Id into grouping
from t in grouping.DefaultIfEmpty()
where school.Id == someSchoolId
select school;
The sql produced is:
SELECT [school].[Id], [school].[StudentId], [school].[TeacherId], [teacher].[Id]
FROM [Schools] AS [school]
LEFT JOIN [Teachers] AS [teacher] ON [school].[TeacherId] = [teacher].[Id]
WHERE [school].[Id] = #__someSchoolId_0
ORDER BY [school].[TeacherId]
But(!), when i try to add one more table to left join
ctx.Schools
.GroupJoin(ctx.Teachers, school => school.TeacherId, teacher => teacher.Id,
(school, teachers) => new { school, teachers })
.SelectMany(info => info.teachers.DefaultIfEmpty(),
(info, teacher) => new { info.school, teacher })
.GroupJoin(ctx.Students, info => info.school.StudentId, student => student.Id,
(info, students) => new {info.school, info.teacher, students})
.SelectMany(info => info.students.DefaultIfEmpty(),
(info, student) => new {info.school, info.teacher, student})
.Where(data => data.school.Id == someSchoolId)
.Select(r => r.school);
or
from school in ctx.Schools
join teacher in ctx.Teachers on school.TeacherId equals teacher.Id into grouping
from t in grouping.DefaultIfEmpty()
join student in ctx.Students on school.StudentId equals student.Id into grouping2
from s in grouping2.DefaultIfEmpty()
where school.Id == someSchoolId
select school;
Threre two separate sql queries produced:
SELECT [student].[Id]
FROM [Students] AS [student]
SELECT [school].[Id], [school].[StudentId], [school].[TeacherId], [teacher].[Id]
FROM [Schools] AS [school]
LEFT JOIN [Teachers] AS [teacher] ON [school].[TeacherId] = [teacher].[Id]
WHERE [school].[Id] = #__someSchoolId_0
ORDER BY [school].[TeacherId]
Looks like there are client-side left join appears.
What am i doing wrong?
You need to select from all the 3 tables so that the left joins would make sense when the Entity Framework translates from the Linq AST to SQL
select new { school, t, s };
instead of
select school;
Then, if you check in Debug from Visual Studio during the program execution and you copy the value of the query to the clipboard, you'll find - as expected - 2 LEFT OUTER JOINs after the FROM
Errata Corrige
The 2 left outer joins were visible from EF 6.
EF Core logger writes that the query ...
could not be translated and will be evaluated locally.
The only remark here is that - without selecting the other tables - there would have been no reasons to find multiple left joins in the first place
EF Core design
Based on the unit testing seen in the github repo and trying to closer meet the OP requirements, I would suggest the following query
var querySO = ctx.Schools
.Include(x => x.Student)
.Include(x => x.Teacher)
;
var results = querySO.ToArray();
This time I see a couple of LEFT JOIN from EF Core Logger
PRAGMA foreign_keys=ON;Executed DbCommand (0ms) [Parameters=[],
CommandType='Text', CommandTimeout='30']
SELECT "x"."SchoolId",
"x"."StudentId", "x"."TeacherId", "s"."StudentId", "s"."name",
"t"."TeacherId", "t"."name"
FROM "Schools" AS "x"
LEFT JOIN "Students"
AS "s" ON "x"."StudentId" = "s"."StudentId"
LEFT JOIN "Teachers" AS
"t" ON "x"."TeacherId" = "t"."TeacherId"
having defined a model
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<School>().HasKey(p => p.SchoolId);
modelBuilder.Entity<Teacher>().HasKey(p => p.TeacherId);
modelBuilder.Entity<Student>().HasKey(p => p.StudentId);
modelBuilder.Entity<School>().HasOne<Student>(s => s.Student)
.WithOne().HasForeignKey<School>(s => s.StudentId);
modelBuilder.Entity<School>().HasOne<Teacher>(s => s.Teacher)
.WithOne().HasForeignKey<School>(s => s.TeacherId);
}
and classes
public class School
{
public long SchoolId { get; set; }
public long? StudentId { get; set; }
public Student Student { get; set; }
public Teacher Teacher { get; set; }
public long? TeacherId { get; set; }
}
public class Student
{
public long StudentId { get; set; }
public string name { get; set; }
}
public class Teacher
{
public long TeacherId { get; set; }
public string name { get; set; }
}