We Keep Coding

LINQ query dropping includes when adding `.Contains()` in where clause

I have a somewhat complex query I'm trying to build in Linq (EntityFramework Core 2.1), and I hit behavior I can't comprehend. The below query runs well and seemingly efficiently:
var q = (
from n in TaskUpdates.Include(t => t.Status).Include("Task").Include("Task.Requirement").Include("User").Include("User.Employee")
where n.User.Employee.EmployeeNumber == 765448466
group n by n.UpdateDate into tu
select tu.OrderByDescending(t=>t.UpdateDate).FirstOrDefault()
)
.Select(x => x.Task.Requirement);
This works as I'd expect, does all the joins I want and includes the expected fields in the SELECT clause:
SELECT [t].[TaskUpdateID], [t].[Active], [t].[TaskId], [t].[Notes], [t].[StatusId], [t].[UpdateDate], [t].[UserId], [t.Task].[TaskID], [t.Task].[Active], [t.Task].[CreatedDate], [t.Task].[RequirementId], [t.Task].[UserId], [t.Task.Requirement].[RequirementID], [t.Task.Requirement].[Active], [t.Task.Requirement].[Description], [t.Task.Requirement].[Hours], [t.Task.Requirement].[Link], [t.Task.Requirement].[Name], [t.Task.Requirement].[RequirementTypeId], [t.Task.Requirement].[ExternalId], [t.Task.Requirement].[SortOrder], [t.Status].[StatusId], [t.Status].[Active], [t.Status].[IsComplete], [t.Status].[Title], [t.User].[UserId], [t.User].[Active], [t.User].[Created], [t.User].[EmployeeNumber], [t.User].[LastLogin], [t.User].[LastUpdated], [t.User.Employee].[EMPLOYEENUMBER], [t.User.Employee].[BEGINDATE], [t.User.Employee].[CITY], [t.User.Employee].[EMPLOYEETYPE], [t.User.Employee].[ENDDATE], [t.User.Employee].[FIRST_NAME], [t.User.Employee].[GENERATION_SUFFIX], [t.User.Employee].[STATUS], [t.User.Employee].[LAST_NAME], [t.User.Employee].[MIDDLE_NAME], [t.User.Employee].[MOBILE], [t.User.Employee].[ORGCODE], [t.User.Employee].[PHONE_NUMBER], [t.User.Employee].[PRIMARYEMAIL], [t.User.Employee].[STATE], [t.User.Employee].[STREET], [t.User.Employee].[TITLE], [t.User.Employee].[ZIPCODE], [t.User.Employee].[BUILDING], [t.User.Employee].[ROOM]
FROM [TaskUpdates] AS [t]
INNER JOIN [Tasks] AS [t.Task] ON [t].[TaskId] = [t.Task].[TaskID]
LEFT JOIN [Requirements] AS [t.Task.Requirement] ON [t.Task].[RequirementId] = [t.Task.Requirement].[RequirementID]
INNER JOIN [Status] AS [t.Status] ON [t].[StatusId] = [t.Status].[StatusId]
INNER JOIN [Users] AS [t.User] ON [t].[UserId] = [t.User].[UserId]
INNER JOIN [DirectoryPeople] AS [t.User.Employee] ON [t.User].[EmployeeNumber] = [t.User.Employee].[EMPLOYEENUMBER]
WHERE [t.User.Employee].[EMPLOYEENUMBER] = 765448466
ORDER BY [t].[UpdateDate]
GO
(I'm using LINQPad to experiment with this query and get the SQL.) In particular, the ending .Select(...) method correctly returns the Requirement object from the query.
What baffles me is if I want to make this query return data for multiple employees, and I change the where clause like so:
var employeeNumbers = new int[] { 765448466 };
var q = (
from n in TaskUpdates.Include(t => t.Status).Include("Task").Include("Task.Requirement").Include("User").Include("User.Employee")
//where n.User.Employee.EmployeeNumber == 765448466
where employeeNumbers.Contains(n.User.Employee.EmployeeNumber)
group n by n.UpdateDate into tu
select tu.OrderByDescending(t=>t.UpdateDate).FirstOrDefault()
)
.Select(x => x.Task.Requirement);
This changes the resulting SQL WHERE clause exactly as I would expect, but it now completely ignores the Includes in the from clause:
SELECT [t].[TaskUpdateID], [t].[Active], [t].[TaskId], [t].[Notes], [t].[StatusId], [t].[UpdateDate], [t].[UserId]
FROM [TaskUpdates] AS [t]
INNER JOIN [Users] AS [t.User] ON [t].[UserId] = [t.User].[UserId]
INNER JOIN [DirectoryPeople] AS [t.User.Employee] ON [t.User].[EmployeeNumber] = [t.User.Employee].[EMPLOYEENUMBER]
WHERE [t.User.Employee].[EMPLOYEENUMBER] IN (765448466)
ORDER BY [t].[UpdateDate]
GO
(only joins as necessary to execute the where) and the result of the final .Select(...) now returns null.
Is this known behavior, with or without explanation? Am I using the Include directives incorrectly, or is there a better way/place for them to go that will resolve this issue?

I can't say for certain the cause, I would suspect EF is going down a different translation path with the Contains and missing the Includes, however as you can see it's not translating the GroupBy at all, so it can definitely be reworked to match more the EF style.
TaskUpdates
.Include(x => x.Task)
.ThenInclude(x => x.Requirement)
.Where(x => employeeNumbers.Contains(x.User.Employee.EmployeeNumber))
.ToList()
.GroupBy(x => x.UpdateDate)
.Select(x => new {
UpdateDate = x.Key,
FirstRequirement = x.First().Task.Requirement
})
.ToList();
This should translate the statements before the first ToList into SQL, populate the results in-memory and allow C# to do the groupby and aggregates on the whole object which SQL would be unable to do.

Linq - how to filter dataset based on data in another table?

I'm looking to filter the data in a dataset much like you'd do a where <value> in (select <value> from other_table where year=2016)
So I have a list of the "values":
var BUs = (from b in dc.BusinessUnits
where b.Year == int.Parse(ddlYears.SelectedValue)
orderby b.BuName
select new { b.BUID }).ToList();
So what I need to do is filter this dataset based on the BUID list returned in the BUs var.
IQueryable<Market> markets = (from p in dc.Markets
orderby p.MarketName
select p);
Help? I'm 100% new to linq so I need a concise solution.

Well if you Market entity has a BUID property and this property it's a primitive type (int, string..) or an enum, you can use Contains method:
var BUs = (from b in dc.BusinessUnits
where b.Year == int.Parse(ddlYears.SelectedValue)
orderby b.BuName
select b.BUID );
IQueryable<Market> markets = (from p in dc.Markets
where BUs.Contains(p.BUID)
orderby p.MarketName
select p);

The standard way of filtering by in memory id list is to use Enumerable.Contains method. But you need first to make sure your list contains ids - they way you wrote it it will contain anonymous type with a property called BUID, by changing the first query like this
int year = int.Parse(ddlYears.SelectedValue);
var BUIDs = (from b in dc.BusinessUnits
where b.Year == year
orderby b.BuName
select b.BUID).ToList();
and then use
var markets = (from p in dc.Markets
where BUIDs.Contains(p.BUID)
orderby p.MarketName
select p);
But note that this will be inefficient. Much better option would be to not use the list of BUIDs for filtering, but combining the 2 queries so the whole thing becomes a single query executed in the database, like this
var markets = (from p in dc.Markets
where dc.BusinessUnits.Any(bu => b.Year == year && b.BUID == p.BUID)
orderby p.MarketName
select p);
This is the exact equivalent of, if using your words, much like you'd do a "where in (select from other_table where year=2016)".

sub linq query is making this take a very long time, how can I make this faster?

I have a list of employees that I build like this:
var employees = db.employees.Where(e => e.isActive == true).ToList();
var latestSales = from es in db.employee_sales.Where(x => x.returned == false);
Now what I want is a result like this:
int employeeId
List<DateTime> lastSaleDates
So I tried this, but the query takes a very very long time to finish:
var result =
(from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates =
(from lsd in latestSales.Where(x => x.EmployeeId == e.EmployeeId)
.Select(x => x.SaleDate)
select lsd).ToList()
};
The above works, but literally takes 1 minute to finish.
What is a more effecient way to do this?

You can use join to get all data in single query
var result = from e in db.employees.Where(x => x.isActive)
join es in db.employee_sales.Where(x => x.returned)
on e.EmployeeId equals es.EmployeeId into g
select new {
EmployeeId = e.employeeId,
LastSaleDates = g.Select(x => x.SaleDate)
};
Unfortunately you can't use ToList() method with Linq to Entities. So either map anonymous objects manually to your EmployeeDetails or change LastSalesDates type to IEnumerable<DateTime>.

Your calls to ToList are pulling things into memory. You should opt to build up a Linq expression instead of pulling an entire query into memory. In your second query, you are issuing a new query for each employee, since your are then operating in the Linq-to-objects domain (as opposed to in the EF). Try removing your calls to ToList.
You should also look into using Foreign Key Association Properties to makes this query a lot nicer. Association properties are some of the most powerful and useful parts of EF. Read more about them here. If you have the proper association properties, your query can look as nice as this:
var result = from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates = e.AssociatedSales
}
You might also consider using a join instead. Read about Linq's Join method here.

Is there an association in your model between employees and latestSales? Have you checked SQL Profiler or other profiling tools to see the SQL that's generated? Make sure the ToList() isn't issuing a separate query for each employee.
If you can live with a result structure as IEnumerable<EmployeeId, IEnumerable<DateTime>>, you could consider modifying this to be:
var result = (from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates = (from lsd in latestSales
where e.employeeId equals lsd.EmployeeId
select lsd.SaleDate)
};
I have some more general recommendations at http://www.thinqlinq.com/Post.aspx/Title/LINQ-to-Database-Performance-hints to help track issues down.

Marc Gravell's Dynamic OrderBy works in one case but not in other

I am trying to do a dynamic order by on columns using Marc Gravell's code. I am posting the 2 queries. It works in one case but doesn't work in 2nd case. Can anybody tell me what changes I need to make to make both queries run perfectly?
This is the link to the Marc Gravell's answer:
https://stackoverflow.com/a/233505
I am using Northwind database. These are both my queries:
var query = (from cust in northwindEntities.Customers
select new
{
City = cust.City ,
Orders = northwindEntities.Orders
.Where(o => o.CustomerID == cust.CustomerID)
.OrderBy("OrderID")
}); // doesn't work.
var query = (from cust in northwindEntities.Customers
select new
{
City = cust.City ,
//Orders = northwindEntities.Orders.Where(o => o.CustomerID == cust.CustomerID).
// OrderBy("OrderID")
}).OrderBy("City"); // works
Here is the exception of the 1st query:
LINQ to Entities does not recognize the method
'System.Linq.IOrderedQueryable1[ConsoleApplication12.Order]
OrderBy[Order](System.Linq.IQueryable1[ConsoleApplication12.Order],
System.String)' method, and this method cannot be translated into a
store expression.

You need to order the final set of result, like you do it in the second case. In first case you order only northwindEntities.Orders.Where( result and not final one.
The correct query is the second.

Obviously it would not work because of the same reason as
var query = (from cust in northwindEntities.Customers
select new
{
City = cust.City ,
Orders = northwindEntities.Orders
.MyCustomMethod()
});
will not work. LINQ-to-Entities will walk through this expression tree and try to convert it to SQL. It can work on known sub set of methods to translate to SQL.
But in the second query, custom OrderBy method dynamically creates the OrderBy that LINQ-to-Entities knows.

How to Join 2 Generic IEnumerators

I'm wondering if its possible to join together IEnumerable's.
Basically I have a bunch of users and need to get their content from the database so I can search and page through it.
I'm using LINQ to SQL, my code at the moment it:
public IEnumerable<content> allcontent;
//Get users friends
IEnumerable<relationship> friends = from f in db.relationships
where f.userId == int.Parse(userId)
select f;
IEnumerable<relationship> freindData = friends.ToList();
foreach (relationship r in freindData)
{
IEnumerable<content> content = from c in db.contents
where c.userId == r.userId
orderby c.contentDate descending
select c;
// This is where I need to merge everything together
}
I hope that make some sense!
Matt

If I understand correctly what you are trying to do, why don't you try doing:
var result = from r in db.relationships
from c in db.contents
where r.userId == int.Parse(userId)
where c.userId == r.UserId
orderby c.contentDate descending
select new {
Relationship = r,
Content = c
}
This will give you an IEnumerable<T> where T is an anonymous type that has fields Relationship and Content.

If you know your users will have less than 2100 friends, you could send the keys from the data you already loaded back into the database easily:
List<int> friendIds = friendData
.Select(r => r.UserId)
.Distinct()
.ToList();
List<content> result = db.contents
.Where(c => friendIds.Contains(c.userId))
.ToList();
What happens here is that Linq translates each Id into a parameter and then builds an IN clause to do the filtering. 2100 is the maximum number of parameters that SQL server will accept... if you have more than 2100 friends, you'll have to break the ID list up and combine (Concat) the result lists.
Or, if you want a more literal answer to your question - Concat is a method that combines 2 IEnumerables together by creating a new IEnumerable which returns the items from the first and then the items from the second.
IEnumerable<content> results = Enumerable.Empty<content>();
foreach (relationship r in friendData)
{
IEnumerable<content> content = GetData(r);
results = results.Concat(content);
}

If you're doing an INNER join, look at the .Intersect() extension method.

Which things are you merging?
There are two main options you could use: .SelectMany(...) or .Concat(...)