We Keep Coding

Is there any possible way to join 2 tables from 2 different databases that use 2 different DbContexts?

So, I'm trying to run the following queries:
var qarDevicesAffected = from a in context.Alerts
join d in context.Devices on a.ResourceId equals d.Id
where !d.Inactive && !a.Inactive && a.Name.Equals(alertName)
select new AffectedEntity.Device
{
Id = a.Id,
DeviceId = d.Id,
TenantId = d.TenantId,
OperatorName = context.Operators
.Select(o => new { o.Id, o.Name })
.First(o => o.Id == d.TenantId).Name,
DeviceSerialNumber = d.SerialNumber,
CellModuleId = d.CellModule.Id,
CellModuleSerialNumber = d.CellModule.SerialNumber,
LastCheckIn = d.LastCheckIn,
LastDownload = d.LastDownload,
LastPackageCreation = d.LastPackageCreation,
TailNumber = d.Tail.TailNumber,
TailId = d.Tail.Id,
CreationDate = oldDate == "CheckIn" ? d.LastCheckIn.Value : role == "LastDownload" ? d.LastDownload.Value : d.LastPackageCreation.Value,
IsPinned = parameters.PinnedIds != null && parameters.PinnedIds.Contains(d.TenantId)
};
var avWifiDevicesAffected = from a in context.Alerts
join d in dataSyncDbContext.Device on a.ResourceId equals d.DeviceId
join p in dataSyncDbContext.PostedFile on a.ResourceId equals p.DeviceId
join c in dataSyncDbContext.CheckIn on a.ResourceId equals c.DeviceId
where d.Status != "ACTIVE" && !a.Inactive && a.Name.Equals(alertName)
select new AffectedEntity.Device
{
Id = a.Id,
DeviceId = d.DeviceId,
TenantId = a.TenantId,
OperatorName = context.Operators
.Select(o => new { o.Id, o.Name })
.First(o => o.Id == a.TenantId).Name,
DeviceSerialNumber = d.SerialNumber,
LastCheckIn = c.UpdatedDate,
LastDownload = p.CreatedDate,
TailId = Guid.Parse(d.TailId)
};
var devicesAffected = qarDevicesAffected.Concat(avWifiDevicesAffected);
And I end up getting the following error:
System.InvalidOperationException:
Cannot use multiple DbContext instances within a single query
execution. Ensure the query uses a single context instance.
It seems obvious that EF Core just does not support this, so, is there a way around using queries that would get me the same result? (Join 2 tables from 2 different dbContexts)

EF Core is an ORM. It wraps raw sql and object mapping behind a nice interface. You can use its Linq-to-Entities functionality to construct a query, which will be converted to sql. However, with two different databases and db contexts, EF core has no idea how to combine them to one sql statement.
If the databases are on one server, you'll have to manually (sql) join the tables and you can use something like Dapper for lower level ORM.
If the databases are on separate server, you must see why EF Core is having a hard time, as there is no simple solution. The relatively simple solution would be to have separate queries and join them in memory. This could have a big performance impact, but there is no good alternative. You can switch from Linq-to-entities (IQueryable) to normal Linq (IEnumerable) by calling .AsEnumerable(). (There also are async options, but that's a more complex story)
In your case, I would split it up into several queries, fetching the exact data you want. Instead of big expensive joins, that just fetch a lot of data. I.e. you first get the Alerts data with the appropriate filter, and then get the appropriate columns in the other table based on matching alert.ResourceId. You can do that by putting the resourdeids together in an array, and filter the tables with with something like
.Where(x => resourceIds.Contains(x.DeviceId)
You will finally have to do some object mapping yourself

Multiple conditions in joins with Linq query

How to apply Multiple conditions in joins and in operator in linq query. I tried to implement the below code and got strucked. Kindly let me know to implement.
Query:
SELECT now() as "time", COALESCE (sum(inv.total_invoice_amount),0) as value1, loc.location_name as metric FROM location loc
LEFT JOIN location_user_map LUM ON LUM.location_id = loc.location_id
LEFT OUTER JOIN invoice inv on inv.client_id IN($client_ids) AND inv.location_id = loc.location_id AND $__timeFilter(inv.end_time)
AND inv.status IN (SELECT status_id FROM status WHERE status IN('Paid','Partialy Paid','Open', 'Completed'))
WHERE loc.client_id IN($client_ids) AND loc.location_name NOT IN('Local Purchase') AND loc.location_id != 0 AND LUM.user_id IN($user_ids)
AND inv.is_active = true
GROUP BY loc.location_name
ORDER BY value1 desc
Code:
using (TransactionContext oTransactionContext = new TransactionContext(iClientID, true))
{
var oPT_Det = (from loc in oTransactionContext.Location
join lum in oTransactionContext.LocationUserMap on loc.LocationId equals lum.LocationId
join inv in oTransactionContext.Invoice on new { loc.LocationId } equals new { inv.LocationId }
select loc);
return oPT_Det;
}

It's not supposed to be this hard, by the way; part of the magic of EF is that it knows how your entities link together and it will do the joins for you. The idea is to use it in the necessary ways for it to write the SQL for you, rather than bring your SQL into C#, jiggle the order and syntax a bit:
var statusIds = context.Statuses
.Where(s => new[]{"Paid","Partialy Paid","Open", "Completed"}.Contains(s.Status))
.Select(s => s.StatusId)
.ToArray();
context.Locations
.Where(l =>
clientIdsArray.Contains(l.ClientId) &&
l.Name != "Local Purchase" &&
l.LocationId != 0 &&
userIdsArray.Contains(l.LocationUserMap)
)
.Select(l => new {
l.LocationId,
l.Invoices.Where(i =>
clientIdsArray.Contains(i.ClientId) &&
statusIds.Contains(I.StatusId)
).Sum(i.TotalInvoiceAmount)
});
Or, perhaps you would start from Invoices instead of locations. It can be easier to start from a many end because the navigation to a one-end is a single property rather than a collection:
context.Invoices.Where(i =>
i.LocationId != 0 &&
i.Location.Name != "Local Purchse" &&
clientIdsArray.Contains(i.Location.ClientId) &&
statusIds.Contains(i.StatusId) &&
i.Location.UserMaps.Any(um => userMapIds.Contains(um.UserId))
)
.GroupBy(i => i.Location.Name)
.Select(g => new { Name = g.Key, Tot = g.Sum(i => i.TotalInvoiceAmount))
EF strives to allow you to just manipulate the entity graph as though it were a local thing, and it manages the DB side for you. Sure, sometimes you have to structure things in a certain way to get the results you want or to get it to craft an SQL in a particular way but..
Note that I don't guarantee these queries as written here solve your problem, or even work/compile; there's a relative lack of info on your question and I've made some assumptions (declared) about your relationships. The purpose of this answer is to point out that you can/are supposed to "leave the SQL at the door" when you come into using EF, rather than thinking of everything in SQL terms still and bending your C# approach to SQL ways. It's intended to be rare that we write the word "join" when working with EF

LINQ query dropping includes when adding `.Contains()` in where clause

I have a somewhat complex query I'm trying to build in Linq (EntityFramework Core 2.1), and I hit behavior I can't comprehend. The below query runs well and seemingly efficiently:
var q = (
from n in TaskUpdates.Include(t => t.Status).Include("Task").Include("Task.Requirement").Include("User").Include("User.Employee")
where n.User.Employee.EmployeeNumber == 765448466
group n by n.UpdateDate into tu
select tu.OrderByDescending(t=>t.UpdateDate).FirstOrDefault()
)
.Select(x => x.Task.Requirement);
This works as I'd expect, does all the joins I want and includes the expected fields in the SELECT clause:
SELECT [t].[TaskUpdateID], [t].[Active], [t].[TaskId], [t].[Notes], [t].[StatusId], [t].[UpdateDate], [t].[UserId], [t.Task].[TaskID], [t.Task].[Active], [t.Task].[CreatedDate], [t.Task].[RequirementId], [t.Task].[UserId], [t.Task.Requirement].[RequirementID], [t.Task.Requirement].[Active], [t.Task.Requirement].[Description], [t.Task.Requirement].[Hours], [t.Task.Requirement].[Link], [t.Task.Requirement].[Name], [t.Task.Requirement].[RequirementTypeId], [t.Task.Requirement].[ExternalId], [t.Task.Requirement].[SortOrder], [t.Status].[StatusId], [t.Status].[Active], [t.Status].[IsComplete], [t.Status].[Title], [t.User].[UserId], [t.User].[Active], [t.User].[Created], [t.User].[EmployeeNumber], [t.User].[LastLogin], [t.User].[LastUpdated], [t.User.Employee].[EMPLOYEENUMBER], [t.User.Employee].[BEGINDATE], [t.User.Employee].[CITY], [t.User.Employee].[EMPLOYEETYPE], [t.User.Employee].[ENDDATE], [t.User.Employee].[FIRST_NAME], [t.User.Employee].[GENERATION_SUFFIX], [t.User.Employee].[STATUS], [t.User.Employee].[LAST_NAME], [t.User.Employee].[MIDDLE_NAME], [t.User.Employee].[MOBILE], [t.User.Employee].[ORGCODE], [t.User.Employee].[PHONE_NUMBER], [t.User.Employee].[PRIMARYEMAIL], [t.User.Employee].[STATE], [t.User.Employee].[STREET], [t.User.Employee].[TITLE], [t.User.Employee].[ZIPCODE], [t.User.Employee].[BUILDING], [t.User.Employee].[ROOM]
FROM [TaskUpdates] AS [t]
INNER JOIN [Tasks] AS [t.Task] ON [t].[TaskId] = [t.Task].[TaskID]
LEFT JOIN [Requirements] AS [t.Task.Requirement] ON [t.Task].[RequirementId] = [t.Task.Requirement].[RequirementID]
INNER JOIN [Status] AS [t.Status] ON [t].[StatusId] = [t.Status].[StatusId]
INNER JOIN [Users] AS [t.User] ON [t].[UserId] = [t.User].[UserId]
INNER JOIN [DirectoryPeople] AS [t.User.Employee] ON [t.User].[EmployeeNumber] = [t.User.Employee].[EMPLOYEENUMBER]
WHERE [t.User.Employee].[EMPLOYEENUMBER] = 765448466
ORDER BY [t].[UpdateDate]
GO
(I'm using LINQPad to experiment with this query and get the SQL.) In particular, the ending .Select(...) method correctly returns the Requirement object from the query.
What baffles me is if I want to make this query return data for multiple employees, and I change the where clause like so:
var employeeNumbers = new int[] { 765448466 };
var q = (
from n in TaskUpdates.Include(t => t.Status).Include("Task").Include("Task.Requirement").Include("User").Include("User.Employee")
//where n.User.Employee.EmployeeNumber == 765448466
where employeeNumbers.Contains(n.User.Employee.EmployeeNumber)
group n by n.UpdateDate into tu
select tu.OrderByDescending(t=>t.UpdateDate).FirstOrDefault()
)
.Select(x => x.Task.Requirement);
This changes the resulting SQL WHERE clause exactly as I would expect, but it now completely ignores the Includes in the from clause:
SELECT [t].[TaskUpdateID], [t].[Active], [t].[TaskId], [t].[Notes], [t].[StatusId], [t].[UpdateDate], [t].[UserId]
FROM [TaskUpdates] AS [t]
INNER JOIN [Users] AS [t.User] ON [t].[UserId] = [t.User].[UserId]
INNER JOIN [DirectoryPeople] AS [t.User.Employee] ON [t.User].[EmployeeNumber] = [t.User.Employee].[EMPLOYEENUMBER]
WHERE [t.User.Employee].[EMPLOYEENUMBER] IN (765448466)
ORDER BY [t].[UpdateDate]
GO
(only joins as necessary to execute the where) and the result of the final .Select(...) now returns null.
Is this known behavior, with or without explanation? Am I using the Include directives incorrectly, or is there a better way/place for them to go that will resolve this issue?

I can't say for certain the cause, I would suspect EF is going down a different translation path with the Contains and missing the Includes, however as you can see it's not translating the GroupBy at all, so it can definitely be reworked to match more the EF style.
TaskUpdates
.Include(x => x.Task)
.ThenInclude(x => x.Requirement)
.Where(x => employeeNumbers.Contains(x.User.Employee.EmployeeNumber))
.ToList()
.GroupBy(x => x.UpdateDate)
.Select(x => new {
UpdateDate = x.Key,
FirstRequirement = x.First().Task.Requirement
})
.ToList();
This should translate the statements before the first ToList into SQL, populate the results in-memory and allow C# to do the groupby and aggregates on the whole object which SQL would be unable to do.

sub linq query is making this take a very long time, how can I make this faster?

I have a list of employees that I build like this:
var employees = db.employees.Where(e => e.isActive == true).ToList();
var latestSales = from es in db.employee_sales.Where(x => x.returned == false);
Now what I want is a result like this:
int employeeId
List<DateTime> lastSaleDates
So I tried this, but the query takes a very very long time to finish:
var result =
(from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates =
(from lsd in latestSales.Where(x => x.EmployeeId == e.EmployeeId)
.Select(x => x.SaleDate)
select lsd).ToList()
};
The above works, but literally takes 1 minute to finish.
What is a more effecient way to do this?

You can use join to get all data in single query
var result = from e in db.employees.Where(x => x.isActive)
join es in db.employee_sales.Where(x => x.returned)
on e.EmployeeId equals es.EmployeeId into g
select new {
EmployeeId = e.employeeId,
LastSaleDates = g.Select(x => x.SaleDate)
};
Unfortunately you can't use ToList() method with Linq to Entities. So either map anonymous objects manually to your EmployeeDetails or change LastSalesDates type to IEnumerable<DateTime>.

Your calls to ToList are pulling things into memory. You should opt to build up a Linq expression instead of pulling an entire query into memory. In your second query, you are issuing a new query for each employee, since your are then operating in the Linq-to-objects domain (as opposed to in the EF). Try removing your calls to ToList.
You should also look into using Foreign Key Association Properties to makes this query a lot nicer. Association properties are some of the most powerful and useful parts of EF. Read more about them here. If you have the proper association properties, your query can look as nice as this:
var result = from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates = e.AssociatedSales
}
You might also consider using a join instead. Read about Linq's Join method here.

Is there an association in your model between employees and latestSales? Have you checked SQL Profiler or other profiling tools to see the SQL that's generated? Make sure the ToList() isn't issuing a separate query for each employee.
If you can live with a result structure as IEnumerable<EmployeeId, IEnumerable<DateTime>>, you could consider modifying this to be:
var result = (from e in employees
select new EmployeeDetails
{
EmployeeId = e.employeeId,
LastSaleDates = (from lsd in latestSales
where e.employeeId equals lsd.EmployeeId
select lsd.SaleDate)
};
I have some more general recommendations at http://www.thinqlinq.com/Post.aspx/Title/LINQ-to-Database-Performance-hints to help track issues down.

Filtering by max and grouping by id with joins to other entities in LINQ to Entity Framework (C#)

The following snippet does work for what I need. I believe though that there must be a better practice? A more optimal way of doing this query?
What is needed is to get a list of employee objects that are the direct reports for employee/mgr x. The direct reports are listed in a history table that has multiple records for each employee, and so only one (the most recent) record should be returned from that table per each direct report (employee) and then the Employee table should be used to get the employee object where employee id is equal to employee id from each history record in this filtered resultset. I can get both halves with two separate LINQ to EF queries.
A problem occurs when trying to join on the employeeHistory object from the first result set. According to MSDN: Referencing Non-Scalar Closures is Not Supported [Referencing a non-scalar closure, such as an entity, in a query is not supported. When such a query executes, a NotSupportedException exception is thrown with a message that states "Unable to create a constant value of type 'Closure type'. Only primitive types ('such as Int32, String, and Guid') are supported in this context."]
So I run two queries and make the first a list of type int rather than a complex object. This does work, but seems contrived. Any suggestions as to a better way (I would like to do one query).
private List<BO.Employee> ListDirectReports(int mgrId)
{
IQueryable<BO.Employee> directRpts;
using(var ctx = new Entities())
{
//to get a list of direct rpts we perform two separate queries. linq to ef with linq to objects
//first one gets a list of emp ids for a direct mgr emp id from the history table
//this first qry uses grouping and a filter by empid and a filter by max(date)
//the second qry joins to the resultset from the first and goes to the employee table
//to get whole employee objects for everyone in the int emp id list from qry #1
//qry #1: just a list of integers (emp ids for those reporting to emp id of mgrId)
IEnumerable<int> directRptIDList =
from employeeHistory in ctx.EmployeeHistory
.Where(h => h.DirectManagerEmployeeID == mgrId).ToList()
group employeeHistory by employeeHistory.EmployeeID into grp
let maxDt = grp.Max(g => g.DateLastUpdated) from history in grp
where history.DateLastUpdated == maxDt
select history.EmployeeID;
//qry #2: a list of Employee objects from the Employee entity. filtered by results from qry #1:
directRpts = from emp in ctx.Employee
join directRptHist in directRptIDList.ToList()
on emp.EmployeeID equals directRptHist
select emp;
}
return directRpts.ToList();
}
Thank you.

2 things I can think of to improve your queries:
ToList is non-deffered. Calling it on your Queryable collections is causing lots of extra trips to the DB. I also believe this call, along with the explicit declaration of IEnumerable<int>, was causing the closure error.
Use the relation between EmployeeHistory and Employee, in your ObjectContex, to join the queries. This will let the Framework produce more efficient SQL. And when directRpts is evaluated on your ToList call, it should only make 1 trip to the DB.
Let me know if this helps.
private List<BO.Employee> ListDirectReports(int mgrId)
{
using(var ctx = new Entities())
{
var directRptIDList =
from employeeHistory in ctx.EmployeeHistory
.Where(h => h.DirectManagerEmployeeID == mgrId)
group employeeHistory by employeeHistory.EmployeeID into grp
let maxDt = grp.Max(g => g.DateLastUpdated) from history in grp
where history.DateLastUpdated == maxDt
select history;
var directRpts =
from emp in ctx.Employee
join directRptHist in directRptIDList
on emp equals directRptHist.Employee
select emp;
}
return directRpts.ToList();
}

There are a number of issues here, not the least of which is that by doing your Where before you get the most recent history item, you're getting records that are no longer valid. Here's how I'd do it:
private List<BO.Employee> ListDirectReports(int mgrId)
{
using(var ctx = new Entities())
{
// First make sure we're only looking at the current employee information
var currentEntries =
from eh in ctx.EmployeeHistory
group employeeHistory by employeeHistory.EmployeeID into grp
select grp.OrderBy(eh => eh.DateLastUpdated).FirstOrDefault();
// Now filter by the manager's ID
var directRpts = currentEntries
.Where(eh => eh.DirectManagerEmployeeID == mgrId);
// This would be ideal, assuming your entity associations are set up right
var employees = directRpts.Select(eh => eh.Employee).Distinct();
// If the above won't work, this is the next-best thing
var employees2 = ctx.Employee.Where(
emp => directRpts.Any(
eh => eh.EmployeeId == emp.EmployeeId));
return employees.ToList();
}
}

Thank you Sorax. The code I had posted did not error and did give me the results I needed, but as you pointed out, merging the two queries errored when including the ToList() method. Using your tip I merged both successfully (tested it) and have posted the improved single query method below. StriplingWarrior I tried yours as well, maybe I could massage it even more. Function evaluation times out on the first query, so I will stick with Sorax' suggestion for now. I appreciate the help and will revisit this.
private static List<BO.Employee> ListDirectReports(int mgrId)
{
IQueryable<BO.Employee> directRpts;
using(var ctx = new Entities())
{
directRpts =
from emp in ctx.Employee
join directRptHist in
(from employeeHistory in ctx.EmployeeHistory
.Where(h => h.DirectManagerEmployeeID == mgrId)
group employeeHistory by employeeHistory.EmployeeID into grp
let maxDt = grp.Max(g => g.DateLastUpdated) from history in grp
where history.DateLastUpdated == maxDt
select history)
on emp equals directRptHist.Employee
select emp;
}
return directRpts.ToList();
//IQueryable<BO.Employee> employees;
//using(var ctx = new Entities())
//{
// //function evaluation times out on this qry:
// // First make sure we're only looking at the current employee information
// IQueryable<BO.EmployeeHistory> currentEntries =
// from eh in ctx.EmployeeHistory
// group eh by eh.EmployeeID into grp
// select grp.OrderBy(eh => eh.DateLastUpdated).FirstOrDefault();
// // Now filter by the manager's ID
// var dirRpts = currentEntries
// .Where(eh => eh.DirectManagerEmployeeID == mgrId);
// // This would be ideal, assuming your entity associations are set up right
// employees = dirRpts.Select(eh => eh.Employee).Distinct();
// //// If the above won't work, this is the next-best thing
// //var employees2 = ctx.Employee.Where(
// // emp => directRpts.Any(
// // eh => eh.EmployeeId == emp.EmployeeId));
//}
//return employees.ToList();
}