We Keep Coding

Entity Framework does not translate Linq expression

I have the following data model, and I need to group list of ResponseItem with these conditions:
First: Group by ResponseItem.Group
Second: Group by ResponseItem.SubGroup, but considering just the most recent one, which means considering the ResponseItem.CreationDate
Code:
public class ResponseItem
{
public string Group { get; set; }
public string SubGroup { get; set; }
public double Value { get; set; }
public DateTime CreationDate { get; set; }
}
public class GroupedResponseItem
{
public string Group { get; set; }
public List<ResponseItem> Items { get; set; }
}
The method is:
public List<GroupedResponseItem> GetGroupedData( IQueryable<ResponseItem> responseItems )
{
return responseItems
.OrderByDescending(i => i.CreationDate)
.GroupBy(i => i.Group)
.Select(grp => new GroupedResponseItem()
{
Group = grp.Key,
Items = grp
.GroupBy(i => new { i.SubGroup })
.Select(grp => grp.First())
.Select(i => new ResponseItem()
{
SubGroup = i.SubGroup,
CreationDate = i.CreationDate,
Value = i.Value
}).ToList()
})
.ToList();
}
But I get an error:
'The LINQ expression 'ProjectionBindingExpression: 0' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'
As I mentioned in the title, I'm using Entity Framework on .NET 6.
On the other hand, If I does not consider the second group by, query works fine:
public List<GroupedResponseItem> GetGroupedData(IQueryable<ResponseItem> responseItems)
{
return responseItems
.OrderByDescending(i => i.CreationDate)
.GroupBy(i => i.Group)
.Select(grp => new GroupedResponseItem()
{
Group = grp.Key,
Items = grp
.Select(i => new ResponseItem()
{
SubGroup = i.SubGroup,
CreationDate = i.CreationDate,
Value = i.Value
})
.ToList()
})
.ToList();
}

The culprit seems to be the secondary projection (Select) here
.GroupBy(i => new { i.SubGroup })
.Select(grp => grp.First()) // <-- (1)
.Select(i => new ResponseItem() // <-- (2)
{
SubGroup = i.SubGroup,
CreationDate = i.CreationDate,
Value = i.Value
})
.ToList()
While EF Core 6.0 has improved translation of GroupBy having additional operators on grouping result set (other than key/aggregates, which have natural SQL support), there are still limitations/defects preventing translation of some constructs. In particular, multiple projections.
Shortly, the Select after GroupBy must be the final LINQ operator. Which is kind of sad, since intermediate projection usually helps the translation and is often used to workaround EF Core limitations. But not in this case.
For this particular query, the projection looks redundant since the type of the elements of the group is the same as the projected type, so it could simply be removed
.GroupBy(i => new { i.SubGroup })
.Select(grp => grp.First()) // <-- final projection
.ToList()
So this is one of the solutions/workarounds. If you really need a projection, because you are selecting partial columns, or project to a different type, then move it inside the Select after GroupBy:
.GroupBy(i => new { i.SubGroup })
.Select(grp => grp
.Select(i => new ResponseItem()
{
SubGroup = i.SubGroup,
CreationDate = i.CreationDate,
Value = i.Value
})
.First()
) // <-- final projection
.ToList()

LINQ, Group by property while keeping other property sort together

So I am not entirely sure how to explain what it is I am trying to do here. I am attempting to take some data (represented by the Excel file screenshot below), and basically sort by Connection2, while keeping similar items in Connection1 together. (Explained a bit in screen shot below)
Here is what I have as of right now:
var wires = RedConductorWires
.OrderBy(x => x.Label)
.ThenBy(x => x.Connection1)
.ThenBy(x => x.Connection2)
.ToList();
Class Object being sorted(Matches Excel Screenshot):
public class CustomExcelFormat
{
public string Label { get; set; }
public string WireSize { get; set; }
public string WireColor { get; set; }
public string WirePartNumber { get; set; }
public string Length { get; set; }
public string Connection1 { get; set; }
public string Connection1Torque { get; set; }
public string Connection1Termination { get; set; }
public string Connection1StripLength { get; set; }
public string Checkbox1 { get; set; }
public string Connection2 { get; set; }
public string Connection2Torque { get; set; }
public string Connection2Termination { get; set; }
public string Connection2StripLength { get; set; }
public string Checkbox2 { get; set; }
}
Screen Shot:
THE PROBLEM:
The issue is if you look at the screen shot the brown "A1:TB7:M1" cells need to be grouped together as well, and the Green "K7:10" need to be grouped together while maintaining their Connection2 sort/group.
In other words, the connection 2 side of those, K8:10 and K8:11 need to stay grouped together.
So obviously my LINQ query is not correct, I believe I need to do some sort of grouping and then sorting but am unsure how to approach it or even ask this question exactly (If someone could put it into words for me). I basically need to group by items in connection 2, while still keeping connection 1 sorted and together.
If someone could point me in the direction of the LINQ expression that could do something like this that would be great!
EDIT
So I used the following query:
var wires = RedConductorWires
.OrderBy(x => x.Label)
.GroupBy(x => new { x.Connection2, x.Connection1 })
.Select(grp => grp.ToList()).SelectMany(i => i).ToList();
and got the grouping correct. Now I just need to get it to sort in some alphabetical manner. See picture below.

Imagine this lines
A - B
C - B
C - D
A - D
you can reorder the lines any way you like and either you would have first column grouped or second column grouped. But you can never have both at the same time

I got the grouping to work correctly with the following query. I decided to keep it sorted on label initially.
var wires = RedConductorWires
.OrderBy(x => x.Label)
.GroupBy(x => new { x.Connection2, x.Connection1 })
.Select(grp => grp.ToList()).SelectMany(i => i).ToList();

Group By the values according to Label, Connection1, Connection2 then sort by these 3 fields and finally the desired output is generated.
var wires = RedConductorWires
.GroupBy(a => new { a.Label,a.Connection2, a.Connection1})
.Join(RedConductorWires,
left=>new { left.Key.Label,left.Key.Connection1,left.Key.Connection2},
right => new { right.Label, right.Connection1, right.Connection2 },
(left,right)=>new {left=left.Key,right = right })
.OrderBy(x => x.left.Label)
.ThenBy(x => x.left.Connection2)
.ThenBy(x => x.left.Connection1)
.ToList();
foreach(var item in wires)
{
Console.WriteLine(item.left.Label + "----" + item.left.Connection1 + "-----" + item.left.Connection2);
}
or
var wires = RedConductorWires
.OrderBy(x => x.Label)
.GroupBy(x => new { x.Connection2, x.Connection1 })
.Select(grp => grp.ToList()).SelectMany(i => i)
.OrderBy(x => x.Connection2)
.ThenBy(x => x.Connection1)
.ToList();
foreach(var item in wires)
{
Console.WriteLine(item.Label + "----" + item.Connection1 + "-----" + item.Connection2);
}

Linq2Db: How to use function Sql.Ext.ListAgg?

I have a model with table Position where are multiple rows with same column value CompanyId and I'd like to group these rows and concat another column Email into comma separated list.
My database is IBM DB2 for i and in SQL I can write query like this:
SELECT
CompanyId,
ListAgg(Email, ',') within group (order by Email) as Emails
FROM Postion
GROUP BY CompanyId
I'm trying to retype the query above to Linq2DB with additional provider LinqToDb4iSeries but without success. The first property GroupedEmails1 builds query without statement "group by", the second one GroupedEmails2 builds nested subquery with group by and parent query with ListAgg - both of them fail to work.
public class Position {
public int CompanyId { get; set; }
public string Email { get; set; }
}
public class MyDataConection : DataConnection {
public ITable<Position> Positions => GetTable<Position>();
public IQueryable<object> GroupedEmails1 => Position
.Select(p => new {
p.CompanyId
Emails = Sql.Ext.ListAgg(p.Email, ",").WithinGroup.OrderBy(p.Email).ToValue()
});
public IQueryable<object> GroupedEmails2 => Position
.GroupBy(p => p.CompanyId)
.SelectMany(g => g.Select(p => new { CompanyId = g.Key, Email = p.Email }))
.Select(p => new {
p.CompanyId
Emails = Sql.Ext.ListAgg(p.Email, ",").WithinGroup.OrderBy(p.Email).ToValue()
});
}
Does anybody have an experience with Linq2Db "Sql.Ext.*" aggregate functions?

I have found the solution for my question above ... maybe it will help someone.
There is an extension method called StringAggregate that is translated into sql function LISTAGG.
public class Position {
public int CompanyId { get; set; }
public string Email { get; set; }
}
public class MyDataConection : DataConnection {
public ITable<Position> Positions => GetTable<Position>();
public IQueryable<object> GroupedEmails1 => Position
.GroupBy(p => p.CompanyId)
.Select(g => new {
CompanyId = g.Key,
Emails = g.StringAggregate(",", x => x.Email).OrderBy(x => x.Email).ToValue()
});
}

I have a LINQ statement that is partially query and partially method. How can I do this using one or the other?

Here is my query as it stands now:
Goals = await (from p in _context.FixtureActivityTb
where p.ActivityType.Trim() == "G"
group p by p.PlayerId into x
join j in _context.PlayerTb on x.Key equals j.PlayerId
select new Stats
{
Name = j.GivenName,
pID = j.PlayerId,
TeamId = j.TeamId,
Count = x.Count()
})
.OrderByDescending(s => s.Count)
.ThenBy(s => s.Name)
.Take(10)
.ToListAsync();
As you can see this is a mix of method and query. I should be able to do a join, where, and add the data to a custom class all in Method, however so far I have not been able to put it all together. Any guidance will be appreciated.
I will include these other items, however, I think they are beside the point.
Variable Declaration:
public IList<Stats> Goals { get; set; }
Class:
public class Stats
{
public Guid pID { get; set; }
public string TeamId { get; set; }
public string Name { get; set; }
public int Count { get; set; }
}

If i understand your question (and the lack of coffee isn't affecting me), to get this all to a Linq chain method, it should be as simple as
Goals = await _context.FixtureActivityTb.Where(p => p.ActivityType.Trim() == "G")
.GroupBy(p => p.PlayerId)
.Join(_context.PlayerTb, x => x.Key, j => j.PlayerId, (x, j)
=> new Stats
{
Name = j.GivenName,
pID = j.PlayerId,
TeamId = j.TeamId,
Count = x.Count()
})
.OrderByDescending(s => s.Count)
.ThenBy(s => s.Name)
.Take(10)
.ToListAsync();

It's quite acceptable to mix the two, especially where you have a base query, and a paging part, as here. You can even compose the queries over multiple statements, eg:
var q = from p in _context.FixtureActivityTb
where p.ActivityType.Trim() == "G"
group p by p.PlayerId into x
join j in _context.PlayerTb on x.Key equals j.PlayerId
select new Stats
{
Name = j.GivenName,
pID = j.PlayerId,
TeamId = j.TeamId,
Count = x.Count()
};
Goals = await q.OrderByDescending(s => s.Count)
.ThenBy(s => s.Name)
.Take(10)
.ToListAsync();

Query with Where before GroupBy

I have an entity like this:
public class Event
{
public string Code;
public DateTimeOffset DateTime;
}
I want to filter by Code and then group by DateTime.Date. I tried this:
var results = session
.Query<Event>()
.Where(e => e.Code == "123")
.GroupBy(e => e.DateTime.Date)
.ToList();
But I get the following error:
Raven.Client.Exceptions.InvalidQueryException: Field 'Code' isn't neither an aggregation operation nor part of the group by key
Query: from Events group by DateTime.Date where Code = $p0
Parameters: {"p0":"123"}
It can be seen from the resulting query that the where clause is being added after the group by clause, which explains the error.
So how do I perform this query in RavenDB?
EDIT:
The code "123" that I used was just an example. I need it to be a variable that is passed to the query, like this:
var results = session
.Query<Event>()
.Where(e => e.Code == code)
.GroupBy(e => e.DateTime.Date)
.ToList();

To start with, learn about the dynamic aggregation query syntax in:
https://demo.ravendb.net/demos/auto-indexes/auto-map-reduce-index
But, in your case you need to define a Static Map-Reduce Index to calculate this for you:(Sum up the number of (filtered) documents per unique Date)
i.e.
public class Result
{
public string Date { get; set; }
public int NumberOfDocs { get; set; }
}
Map = events => from event in events
where event.Code == "123"
select new Result
{
Date = event.DateTime.Date
NumberOfDocs = 1
}
Reduce = results => from result in results
group result by result.Date into g
select new Result
{
Date = g.Key,
Count = g.Sum(x => x.NumberOfDocs )
}
==> Learn about Static Map-Reduce Index in:
https://demo.ravendb.net/demos/static-indexes/map-reduce-index
Follow the detailed Walkthrough..
----------
Update:
You can use the following map-reduce index that aggregates the number of documents per Code & Date 'couple', and then you can query with 'Code'
public class Result
{
public string Code { get; set; }
public string Date { get; set; }
public int NumberOfDocs { get; set; }
}
Map = events => from event in events
select new Result
{
Code = event.Code
Date = event.DateTime.Date
NumberOfDocs = 1
}
Reduce = results => from result in results
group result by new
{
result.Code,
result.Date
}
into g
select new Result
{
Code = g.Key.Code
Date = g.Key.DateTime.Date,
NumberOfDocs = g.Sum(x => x.NumberOfDocs )
}
and then query
List<Result> queryResults = session.Query< Result, <Index_Name> >()
.Where(x => x.Code == "some-code-number")
.ToList();
and then you can also do in your code
queryResults.GroupBy(x => x.Date)