We Keep Coding

Why there is no GroupBy clause in internal SQL of Entity Framework linq query?

In documentation of Entity Framework:
https://www.entityframeworktutorial.net/querying-entity-graph-in-entity-framework.aspx
in section regarding GroupBy we can read that following code:
using (var ctx = new SchoolDBEntities())
{
var students = from s in ctx.Students
group s by s.StandardId into studentsByStandard
select studentsByStandard;
foreach (var groupItem in students)
{
Console.WriteLine(groupItem.Key);
foreach (var stud in groupItem)
{
Console.WriteLine(stud.StudentId);
}
}
}
executes internally following SQL:
SELECT
[Project2].[C1] AS [C1],
[Project2].[StandardId] AS [StandardId],
[Project2].[C2] AS [C2],
[Project2].[StudentID] AS [StudentID],
[Project2].[StudentName] AS [StudentName],
[Project2].[StandardId1] AS [StandardId1]
FROM ( SELECT
[Distinct1].[StandardId] AS [StandardId],
1 AS [C1],
[Extent2].[StudentID] AS [StudentID],
[Extent2].[StudentName] AS [StudentName],
[Extent2].[StandardId] AS [StandardId1],
CASE WHEN ([Extent2].[StudentID] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C2]
FROM (SELECT DISTINCT
[Extent1].[StandardId] AS [StandardId]
FROM [dbo].[Student] AS [Extent1] ) AS [Distinct1]
LEFT OUTER JOIN [dbo].[Student] AS [Extent2] ON ([Distinct1].[StandardId] = [Extent2]. [StandardId]) OR (([Distinct1].[StandardId] IS NULL) AND ([Extent2].[StandardId] IS NULL))
) AS [Project2]
ORDER BY [Project2].[StandardId] ASC, [Project2].[C2] ASC
go
Why there is no GroupBy clause in SQL? If there is no GroupBy clause needed, can’t we just use simple Select with OrderBy and without Joins? Can anyone explain the above query?

The bottom line is: because SQL can't return nested result sets.
Every SQL SELECT statement returns a flat list of values. LINQ is capable of returning object graphs, i.e. objects with nested objects. That's exactly what LINQ's GroupBy does.
In SQL, a GROUP BY statement only returns the grouping columns and aggregate results:
SELECT StandardId, COUNT(*)
FROM Students
GROUP BY StandardId;
The rest of the student columns is gone.
A LINQ GroupBy statement returns something like
StandardId
StudentId StudentName
1
21 "Student1"
15 "Student2"
2
48 "Student3"
91 "Student4"
17 "Student5"
Therefore, a SQL GROUP BY statement can never be the source for a LINQ GroupBy.
Entity Framework (6) knows this and it generates a SQL statement that pulls all required data from the database, adds some parts that make grouping easier, and it creates the groupings client-side.

Linq generate select with no from

In sql I can do this
select 'a' as MyColumn
so, i have a query in linq with entity framework that get some data from the database, but, i need union that query with one row, and in sql I can do this:
select ... from ...
union
select 'a' as MyColumn
How can i generate this query with linq?
I tried to do this:
var query = (from ... select new {..}).Union(new List<...> { new ...() { MyColumn = 'a' } })
But i gess that Entity Framework DON'T know how to translate that in memory list to sql
I need to get an IQueryable result, not a List or other in memory Collection, because i need to join that result to other sql linq querys in the future.

This isn't possible and you shouldn't do it. Both for the same reason: Entity Framework will try to translate the whole LINQ statement into SQL, including the local list (new List<...>).
The reason why it's not possible is that EF has no way to translate C# objects into SQL constructs.
The reason why you shouldn't do it is that it's incredibly wasteful: you build the list in C# code, EF (if it could) translates it into a SQL statement, the database runs the SQL statement and converts it to a result set, EF receives the result set and converts it into the list you originally offered it.
Just to demonstrate it, I'll show what happens if you do this with a list of primitive values which EF does know how to translate into SQL:
var ints = Enumerable.Range(1,5);
var res = Products.Select(c => c.Id).Union(ints).ToList();
This produces the following SQL statement:
SELECT
[Distinct1].[C1] AS [C1]
FROM ( SELECT DISTINCT
[UnionAll5].[ProductId] AS [C1]
FROM (SELECT
[Extent1].[ProductId] AS [ProductId]
FROM [dbo].[Product] AS [Extent1]
UNION ALL
SELECT
1 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
UNION ALL
SELECT
2 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable2]
UNION ALL
SELECT
3 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable3]
UNION ALL
SELECT
4 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable4]
UNION ALL
SELECT
5 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable5]) AS [UnionAll5]
) AS [Distinct1]
As you see, for each element in the list EF generated a SingleRowTablex entry to build a "temp table" to UNION with the ids from the actual query.
Conclusion: just query what you need from the database and add to the result afterwards. It's easy enough to do that:
(from ... select new {..})
.AsEnumerable() // continue in memory
.Union(...)

How does skip and take works in linq

i have following Linq query . its works well but the thing that seems confusing is how does skip() and take() function working in linq.
here is my query
(from GRD in _tblAcademicYears.GetQueryable()
where GRD.SchoolID == intSchoolID
select new AcademicYearsModel
{
AcademicYearID = GRD.AcademicYearID,
SchoolID = GRD.SchoolID,
AcademicYearName = GRD.AcademicYearName,
AcademicYearStart = GRD.AcademicYearStart,
AcademicYearEnd = GRD.AcademicYearEnd,
AcademicYearRemarks = GRD.AcademicYearRemarks,
IsActive = GRD.IsActive,
CreatedOn = GRD.CreatedOn,
CreatedBy = GRD.CreatedBy,
ModifiedOn = GRD.ModifiedOn,
ModifiedBy = GRD.ModifiedBy
}
).Where(z => z.AcademicYearName.Contains(param.sSearch) || z.AcademicYearStart.ToString().Contains(param.sSearch)
|| z.AcademicYearEnd.ToString().Contains(param.sSearch) || z.AcademicYearRemarks.Contains(param.sSearch))
.Skip(param.iDisplayStart).Take(param.iDisplayLength).ToList();
How this query will get record from data base .
will it get all record from database and then will apply skip() and take().
or it will just get record that are with in limits of skip() and take()

When you call .Take only, it will just translate to SQL: TOP N syntax
When you call .Skip and .Take together, it will generate at least 2 queries, by using ROWNUMBER to filter out.
So the short answer for your question is: No, it will not get all records from database. it will run a SQL to filter and select.
If you are curious, you can always use SQL profiler or just check the generated SQL in the debug mode.
Here is a simple MSDN article explains it
https://msdn.microsoft.com/library/bb386988(v=vs.100).aspx

If you asking about LINQ to SQL, you can run a sql-profiler to get query, generated by linq provider.
But I can tell you, LINQ will get only records in limits skip and take, using row_number operator in SQL:
The query will be like this (skip 3 and take 3):
SELECT TOP (3)
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
FROM (
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
row_number() OVER (ORDER BY [Extent1].[Name] ASC) AS [row_number]
FROM [dbo].[tec_Stores] AS [Extent1]
) AS [Extent1]
WHERE [Extent1].[row_number] > 3
ORDER BY [Extent1].[Name] ASC
In LINQ to Entities it works different, depending on collection you use.

The source code of all Linq IEnumerable extensions can be found here:
System.Linq.Enumerable
Here you can see how skip and take work

EntityFramework Group by not included in SQL statement

I'm trying to create a query similar to this:
select randomId
from myView
where ...
group by randomId
NOTE: EF doesn't support the distinct so I was thinking of going around the lack of it with the group by (or so I think)
randomId is numeric
Entity Framework V.6.0.2
This gives me the expected result in < 1 second query
When trying to do the same with EF I have been having some issues.
If I do the LINQ similar to this:
context.myView
.Where(...)
.GroupBy(mt => mt.randomId)
.Select({ Id = group.Key, Count = group.Count() } )
I will get sort of the same result but forcing a count and making the query > 6 seconds
The SQL EF generates is something like this:
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [randomId],
[GroupBy1].[A1] AS [C2]
FROM (
SELECT
[Extent1].[randomId] AS [K1],
COUNT(1) AS [A1]
FROM [dbo].[myView] AS [Extent1]
WHERE (...)
GROUP BY [Extent1].[randomId]
) AS [GroupBy1]
But, if the query had the count commented out it would be back to < 1 second
If I change the Select to be like:
.Select({ Id = group.Key} )
I will get all of rows without the group by statement in the SQL query and no Distinct whatsoever:
SELECT
[Extent1].[anotherField] AS [anotherField], -- 'this field got included automatically on this query and I dont know why, it doesnt affect outcome when removed in SQL server'
[Extent1].[randomId] AS [randomId]
FROM [dbo].[myView] AS [Extent1]
WHERE (...)
Other failed attempts:
query.GroupBy(x => x.randomId).Select(group => group.FirstOrDefault());
The query that was generated is as follows:
SELECT
[Limit1].ALL FIELDS,...
FROM (SELECT
[Extent1].[randomId] AS [randomId]
FROM [dbo].[myView] AS [Extent1]
WHERE (...) AS [Project1]
OUTER APPLY (SELECT TOP (1)
[Extent2].ALL FIELDS,...
FROM [dbo].[myView] AS [Extent2]
WHERE (...) AS [Limit1] -- same as the where above
This query performed rather poorly and still managed to return all Ids for the where clause.
Does anyone have an idea on how to force the usage of the group by without an aggregating function like a count?
In SQL it works but then again I have the distinct keyword as well...
Cheers,
J

var query = from p in TableName
select new {Id = p.ColumnNameId};
var distinctItems = query.Distinct().ToList();
Here is the linq query however you should be able to write an equivalent from EF dbset too. If you have issues let me know.
Cheers!

Linq-to-entities, get results + row count in one query

I've seen multiple questions about this matter, however they were 2 years (or more) old, so I'd like to know if anything changed about this.
The basic idea is to populate a gridview and create custom paging. So, I need the results and row count as well.
In SQL this would be something like:
SELECT COUNT(id), Id, Name... FROM ... WHERE ...
Getting everything in a nice simple query. However, I'd like to be consistent and use Linq2Entities.
So far I'm using the approach with two queries (against sql server), because it just works. I would like to optimize it though and use a single query instead.
I've tried this:
var query = from o in _db.Products
select o;
var prods = from o in query
select new
{
Count = query.Count(),
Products = query
};
This produces a very nasty and long query with really unnecessary cross joins and other stuff which I don't really need or want.
Is there a way to get the paged results + count of all entities in a one simple query? What is the recommended approach here?
UPDATE:
Just tried FutureQueries and either I'm doing something wrong, or it actually executes two queries. This shows my sql profiler:
-- Query #1
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[Products] AS [Extent1]
WHERE 1 = [Extent1].[CategoryID]
) AS [GroupBy1];
And next row:
-- Query #1
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[Price] AS [Price],
[Extent1].[CategoryID] AS [CategoryID]
FROM [dbo].[Products] AS [Extent1]
WHERE 1 = [Extent1].[CategoryID];
The C# code:
internal static List<Product> GetProducts(out int _count)
{
DatabaseEntities _db = new DatabaseEntities();
var query = from o in _db.Products
where o.CategoryID == 1
select o;
var count = query.FutureCount();
_count = count.Value;
return query.Future().ToList();
}
Did I miss something? According to my profiler it does exactly the same except that added row in the query (-- Query #1).

Have a look at Future Queries to do this in EntityFramework.Extended. The second example on that linked page uses FutureCount() to do exactly what you want. Adapted here:
var q = db.Products.Where(p => ...);
var qCount = q.FutureCount();
var qPage = q.Skip((pageNumber-1)*pageSize).Take(pageSize).Future();
int total = qCount.Value; // Both queries are sent to the DB here.
var tasks = qPage.ToList();

this 'EntityFramework.Extended' library is no longer supported use this one instead:
entityframework-plus and go here:
https://entityframework-plus.net/query-future to see how you can get count and records
in the same query.