How to apply Multiple conditions in joins and in operator in linq query. I tried to implement the below code and got strucked. Kindly let me know to implement.
Query:
SELECT now() as "time", COALESCE (sum(inv.total_invoice_amount),0) as value1, loc.location_name as metric FROM location loc
LEFT JOIN location_user_map LUM ON LUM.location_id = loc.location_id
LEFT OUTER JOIN invoice inv on inv.client_id IN($client_ids) AND inv.location_id = loc.location_id AND $__timeFilter(inv.end_time)
AND inv.status IN (SELECT status_id FROM status WHERE status IN('Paid','Partialy Paid','Open', 'Completed'))
WHERE loc.client_id IN($client_ids) AND loc.location_name NOT IN('Local Purchase') AND loc.location_id != 0 AND LUM.user_id IN($user_ids)
AND inv.is_active = true
GROUP BY loc.location_name
ORDER BY value1 desc
Code:
using (TransactionContext oTransactionContext = new TransactionContext(iClientID, true))
{
var oPT_Det = (from loc in oTransactionContext.Location
join lum in oTransactionContext.LocationUserMap on loc.LocationId equals lum.LocationId
join inv in oTransactionContext.Invoice on new { loc.LocationId } equals new { inv.LocationId }
select loc);
return oPT_Det;
}
It's not supposed to be this hard, by the way; part of the magic of EF is that it knows how your entities link together and it will do the joins for you. The idea is to use it in the necessary ways for it to write the SQL for you, rather than bring your SQL into C#, jiggle the order and syntax a bit:
var statusIds = context.Statuses
.Where(s => new[]{"Paid","Partialy Paid","Open", "Completed"}.Contains(s.Status))
.Select(s => s.StatusId)
.ToArray();
context.Locations
.Where(l =>
clientIdsArray.Contains(l.ClientId) &&
l.Name != "Local Purchase" &&
l.LocationId != 0 &&
userIdsArray.Contains(l.LocationUserMap)
)
.Select(l => new {
l.LocationId,
l.Invoices.Where(i =>
clientIdsArray.Contains(i.ClientId) &&
statusIds.Contains(I.StatusId)
).Sum(i.TotalInvoiceAmount)
});
Or, perhaps you would start from Invoices instead of locations. It can be easier to start from a many end because the navigation to a one-end is a single property rather than a collection:
context.Invoices.Where(i =>
i.LocationId != 0 &&
i.Location.Name != "Local Purchse" &&
clientIdsArray.Contains(i.Location.ClientId) &&
statusIds.Contains(i.StatusId) &&
i.Location.UserMaps.Any(um => userMapIds.Contains(um.UserId))
)
.GroupBy(i => i.Location.Name)
.Select(g => new { Name = g.Key, Tot = g.Sum(i => i.TotalInvoiceAmount))
EF strives to allow you to just manipulate the entity graph as though it were a local thing, and it manages the DB side for you. Sure, sometimes you have to structure things in a certain way to get the results you want or to get it to craft an SQL in a particular way but..
Note that I don't guarantee these queries as written here solve your problem, or even work/compile; there's a relative lack of info on your question and I've made some assumptions (declared) about your relationships. The purpose of this answer is to point out that you can/are supposed to "leave the SQL at the door" when you come into using EF, rather than thinking of everything in SQL terms still and bending your C# approach to SQL ways. It's intended to be rare that we write the word "join" when working with EF
Related
So, I'm trying to run the following queries:
var qarDevicesAffected = from a in context.Alerts
join d in context.Devices on a.ResourceId equals d.Id
where !d.Inactive && !a.Inactive && a.Name.Equals(alertName)
select new AffectedEntity.Device
{
Id = a.Id,
DeviceId = d.Id,
TenantId = d.TenantId,
OperatorName = context.Operators
.Select(o => new { o.Id, o.Name })
.First(o => o.Id == d.TenantId).Name,
DeviceSerialNumber = d.SerialNumber,
CellModuleId = d.CellModule.Id,
CellModuleSerialNumber = d.CellModule.SerialNumber,
LastCheckIn = d.LastCheckIn,
LastDownload = d.LastDownload,
LastPackageCreation = d.LastPackageCreation,
TailNumber = d.Tail.TailNumber,
TailId = d.Tail.Id,
CreationDate = oldDate == "CheckIn" ? d.LastCheckIn.Value : role == "LastDownload" ? d.LastDownload.Value : d.LastPackageCreation.Value,
IsPinned = parameters.PinnedIds != null && parameters.PinnedIds.Contains(d.TenantId)
};
var avWifiDevicesAffected = from a in context.Alerts
join d in dataSyncDbContext.Device on a.ResourceId equals d.DeviceId
join p in dataSyncDbContext.PostedFile on a.ResourceId equals p.DeviceId
join c in dataSyncDbContext.CheckIn on a.ResourceId equals c.DeviceId
where d.Status != "ACTIVE" && !a.Inactive && a.Name.Equals(alertName)
select new AffectedEntity.Device
{
Id = a.Id,
DeviceId = d.DeviceId,
TenantId = a.TenantId,
OperatorName = context.Operators
.Select(o => new { o.Id, o.Name })
.First(o => o.Id == a.TenantId).Name,
DeviceSerialNumber = d.SerialNumber,
LastCheckIn = c.UpdatedDate,
LastDownload = p.CreatedDate,
TailId = Guid.Parse(d.TailId)
};
var devicesAffected = qarDevicesAffected.Concat(avWifiDevicesAffected);
And I end up getting the following error:
System.InvalidOperationException:
Cannot use multiple DbContext instances within a single query
execution. Ensure the query uses a single context instance.
It seems obvious that EF Core just does not support this, so, is there a way around using queries that would get me the same result? (Join 2 tables from 2 different dbContexts)
EF Core is an ORM. It wraps raw sql and object mapping behind a nice interface. You can use its Linq-to-Entities functionality to construct a query, which will be converted to sql. However, with two different databases and db contexts, EF core has no idea how to combine them to one sql statement.
If the databases are on one server, you'll have to manually (sql) join the tables and you can use something like Dapper for lower level ORM.
If the databases are on separate server, you must see why EF Core is having a hard time, as there is no simple solution. The relatively simple solution would be to have separate queries and join them in memory. This could have a big performance impact, but there is no good alternative. You can switch from Linq-to-entities (IQueryable) to normal Linq (IEnumerable) by calling .AsEnumerable(). (There also are async options, but that's a more complex story)
In your case, I would split it up into several queries, fetching the exact data you want. Instead of big expensive joins, that just fetch a lot of data. I.e. you first get the Alerts data with the appropriate filter, and then get the appropriate columns in the other table based on matching alert.ResourceId. You can do that by putting the resourdeids together in an array, and filter the tables with with something like
.Where(x => resourceIds.Contains(x.DeviceId)
You will finally have to do some object mapping yourself
The following LINQ query in EF (EFCore, v2.2.1)
var x = context.Exchange
.Include(q => q.Input)
.Where(q => q.InputId != 1&&
q.Input.CreatedOnUtc > DateTime.Parse("2019-11-25") &&
q.Input.UserId == 2 &&
q.BotConversationId == 3)
.Distinct()
.OrderBy(q => q.Input.CreatedOnUtc)
.FirstOrDefault()
Ends up giving the profiled SQL results (simplified)
select * from (
select distinct e.*
from Exchange e, ExchangeInput i
where e.InputId = i.InputId
and e.InputId <> 1
and i.UserId = 2
and e.BotConversationId = 3
)
select * from ExchangeInput
Why does it need to do two separate queries? The second query being horrendous when ExchangeInput might have millions of rows.
Surely, this would suffice:
select * from (
select distinct e.*, i.CreatedOnUtc
from Exchange e, ExchangeInput i
where e.InputId = i.InputId
and e.InputId <> 1
and i.UserId = 2
and e.BotConversationId = 3
) a
order by a.CreatedOnUtc
Also - putting the Distinct after the order by gives only 1 query as I would expect.
Fixing the problem is easy enough. Adding a .Select(...) before the .Distinct or removing the .Distinct() will do it. But the initial, poorly performing code, doesn't seem immediately problematic when reviewing it.
I would start by suggesting that calling Distinct() before a FirstOrDefault() is unnecessary. The first row in a "non-distinct" query should always be the same as a "distinct" query when you have an OrderBy! As you mentioned in your last sentence, it seems that removing the Distinct() should only create one query.
Separate to your question, I would also suggest calculating DateTime.Parse("2019-11-25") outside of the query. That should allow you to pass it to the database server as a parameter and that might make your query even more efficient.
All in all, I would try:
var dateFilter = DateTime.Parse("2019-11-25");
var x = context.Exchange
.Include(q => q.Input)
.Where(q => q.InputId != 1 &&
q.Input.CreatedOnUtc > dateFilter &&
q.Input.UserId == 2 &&
q.BotConversationId == 3)
.OrderBy(q => q.Input.CreatedOnUtc)
.FirstOrDefault()
I am measuring differences in query execution and stumbled upon a case I have no explanation for. The query should retrieve 10000 customers with their main address (a customer can have many addresses). We used 2 different methods with Navigation Properties which differ greatly in execution time.
The first method retrieves the customers the way I usually write Linq queries: write the results directly to a business object and calling ToList(). This method takes 25 seconds to execute.
The second method retrieves the customers as a list of EF Entities first. The EF Entities are converted to business objects in a foreach loop. This method takes 2 seconds to execute.
Can someone explain the difference? And is it possible to modify the first method so the execution time is similar to the second?
private List<ICustomer> NavigationProperties_SO(int method)
{
using (Entities context = new Entities())
{
context.Database.Log = s => System.Diagnostics.Debug.WriteLine(s);
context.Configuration.ProxyCreationEnabled = false;
context.Configuration.AutoDetectChangesEnabled = false;
List<ICustomer> customerList = new List<ICustomer>();
if (method == 1)
{
// Execution time: 25 seconds
customerList = (from c in context.cust
.Include(o => o.AddressList)
.Include(o => o.AddressList.Select(p => p.ADDR))
let mainAddress = c.AddressList.Where(o => o.Main_addr == "1").FirstOrDefault()
select new Customer
{
cust = c,
mainAddress = mainAddress,
addr = mainAddress == null ? null : mainAddress.ADDR
}).AsNoTracking().ToList<ICustomer>();
}
else if (method == 2)
{
// Execution time: 2 seconds
var tempList = (from c in context.cust
.Include(o => o.AddressList)
.Include(o => o.AddressList.Select(p => p.ADDR))
select c).AsNoTracking().ToList();
foreach (var c in tempList)
{
ICustomer customer = new Customer();
var mainaddress = c.AddressList.Where(o => o.Main_addr == "1").FirstOrDefault();
customer.cust = c;
customer.mainAddress = mainaddress;
customer.addr = mainaddress == null ? null : mainaddress.ADDR;
customerList.Add(customer);
}
}
return customerList;
}
}
Edit
Here are the (simplified) queries generated by Entity Framework:
Method 1
SELECT
*
FROM [DBA].[CUST] AS [Extent1]
OUTER APPLY (SELECT TOP ( 1 )
*
FROM [DBA].[CUST_ADDR] AS [Extent2]
WHERE (([Extent1].[Id] = [Extent2].[Id]) AND (N'1' = [Extent2].[Main_addr])
ORDER BY 'a' ) AS [Limit1]
LEFT OUTER JOIN [DBA].[ADDR] AS [Extent3] ON [Limit1].[Id] = [Extent3].[Id]
Method 2
SELECT
*
FROM ( SELECT
*
FROM [DBA].[CUST] AS [Extent1]
LEFT OUTER JOIN (SELECT *
FROM [DBA].[CUST_ADDR] AS [Extent2]
LEFT OUTER JOIN [DBA].[ADDR] AS [Extent3] ON [Extent2].[Id] = [Extent3].[Id] ) AS [Join1] ON ([Extent1].[Id] = [Join1].[Id])
) AS [Project1]
The difference is that the first method does the filtering in the query (´let´) while the second method retrieves all records and filters in the loop.
I suspect
let mainAddress = c.AddressList.Where(o => o.Main_addr == "1").FirstOrDefault()
is the culprit. Certain queries forces EF to ask for all possible combinations to be returned. EF then spends a little time narrowing down the scope before it provides you with a reasonable result set. You can use SQL Server Profiler to look at the queries generated.
In any case, you can use LINQ, rather than a foreach, at the end of your second method (this won't help performance, but readability might improve):
return tempList.Select(c => new Customer{cust=c, mainAddress = c.AddressList.FirstOrDefault(o=>o.Main_addr=="1"), ...);
Answer related to comments... (but two long for a comment)
For the "how to choose the best syntax" part
I would say that it comes partially from "experience" (see, 9Rune5 and I suspected the same point, which was the problematic one before seeing the generated sql) : but experience, sometimes, may also leed to wrong conclusions ;)
So to be a little bit more pragmatic, I would suggest you to use tools/libs which will help you to look at the generated sql / time by query, or page...
ANTS Performance profiler, Miniprofiler, Sql Server profiler, etc, it may depend on your technologies / needs...
By the way, if you want to keep a "linq" syntax, you could go for
var tempList = context.cust
.Include(o => o.AddressList)
.Include(o => o.AddressList.Select(p => p.ADDR))
.AsNoTracking()
.ToList();
var result = (from c in tempList
let mainAddress = c.AddressList.Where(o => o.Main_addr == "1").FirstOrDefault()
select new Customer
{
cust = c,
mainAddress = mainAddress,
addr = mainAddress == null ? null : mainAddress.ADDR
}).ToList<ICustomer>();
But not really less verbose than the foreach syntax...
Something weird is going on.
If I do this:
var allAccountsQuery = from acc in baseQ
where
//high potential check - 1, 2, 3
(acc.mcpl_potencjal_klienta == 1 || acc.mcpl_potencjal_klienta == 2 || acc.mcpl_potencjal_klienta == 3) &&
//directors block check
((acc.mcpl_blokada_dyrektorska == true && acc.mcpl_blokada_do <= date) || acc.mcpl_blokada_dyrektorska == false || acc.mcpl_blokada_dyrektorska == null) &&
//sack assign date check
(acc.mcpl_dataprzypisaniazworka == null || acc.mcpl_dataprzypisaniazworka < date) &&
//owner change check
(acc.mcpl_datazmianywasciciela == null || acc.mcpl_datazmianywasciciela < date) &&
//creation date check
//TODO:For testing!
//(acc.mcpl_data_utworzenia_test < date)
(acc.createdon < date)
select acc;
var joinQuery = from acc in allAccountsQuery
join opp in ctx.opportunityopportunities on acc.accountid equals opp.customerid.Value
select new
{
Account = acc,
Opportunity = opp
};
Plugins.Common.XrmHelper.ClearCache("account");
var joinResult = joinQuery.ToList();
Then I'll get an unknown platform error when executing this query. I need to copy-paste the WHOLE where clause from allAccountsQuery to the joinQuery and use baseQ again, and then it works.
What's going on here? I thought you can safely join LINQ queries as long as you're not doing any unsupported operations.
PS. The STRANGEST part is that the pasted code WILL work with slightly different where conditions.
PPS. baseQ is just an even simpler where query, much like the allAccountsQuery.
Maybe is not the answer but as I can't leave a comment and no one has answer I think this could help.
Why you don't do the join in the first query? As from I know the LINQ CRM queries have problems joining tables when in the clause WHERE we have the OR Predicate, and not when we try to select from different tables, I think for you query should work. I have one post explaining what I learned.
Linq-to-CRM has a limited set of supported operations compared to other providers life EF or Linq-to-SQL.
You may have better success hydrating one or both of the two queries. Since your account query has a where clause try hydrating it:
var joinQuery = from acc in allAccountsQuery.ToList() // call ToList() to hydrate the query
join opp in ctx.opportunityopportunities
on acc.accountid equals opp.customerid.Value
select new
{
Account = acc,
Opportunity = opp
};
If you have a LARGE number of Opportunities you may want to try and filter that query based on the accounts returned from the first query before doing the Join.
I have these working in sql for large data set working great. However I'm having hard time converting to linq-I'm new to linq
Select * from table1 t1, table2 t2 where (t1.RoleId+t1.UserId)!=(t2.RoleId+t2.UserId).
On a side note, when the two variables are separated, I get undesired results.
Meaning the following: where (t1.RoleId != t2.RoleId && t1.UserId != t2.UserId)
In c# I have two anonymous lists. The last linq statement works great till nulls come into the picture. Nothing returns. I even thought of using a left join with no success.
So how would you tackle the above query with anonymous type lists?
Linq statments I have so far
var roleUserList =
(
from rls in roleResouceList
join user in userResourceList
on rls.FullResource.ToUpper() equals user.FullResource.ToUpper()
orderby rls.RoleID, user.UserID, rls.Res1, rls.Res2, rls.Res3
select new
{
RoleID = rls.RoleID,
UserID = user.UserID,
ServerId = rls.ServerID,
FullResource = rls.FullResource,
RlsRes1 = rls.Res1,
RlsRes2 = rls.Res2,
RlsRes3 = rls.Res3
}).Distinct().ToList();
var missingRoleUserList =
(
from rls in rlsCount
join usr in usrCount
on rls.Res1 equals usr.Res1
where rls.Total > usr.Total
select new
{
UserID = usr.UsrID,
RoleID = rls.RoleID
}).Distinct().ToList();
List<string> outputRoleUserList =
(
from rls in roleUserList
from mis in missingRoleUserList
where (rls.RoleID + rls.UserID) != (mis.RoleID +mis.UserID)
select rls.UserID + ",\"" + rls.RoleID
).DefaultIfEmpty().Distinct().ToList();
I'm not entirely certain that this is what you're looking for, but I'm going to give it a shot:
Try chaining your where clauses in Linq to SQL, and you may get a better result:
List<string> outputRoleUserList =
from rls in roleUserList
from mis in missingRoleUserList
where rls.RoleID != mis.RoleID
where rls.UserID != mis.UserID
select rls.UserID + ",\"" + rls.RoleID
This will actually generate SQL as follows:
rls.RoleId != mis.UserID AND rls.UserId != mis.UserID
However, you have already forced execution on roleUserList and missingRoleUserList, so what you're using in the third Linq statement is not really Linq to SQL but rather Linq to Objects, if I'm reading this correctly.
I'd be curious to see some additional information or clarification and then maybe I'll understand better what's going on!
EDIT: I realized another possibility, it's possible that the object.UserID or object.RoleID is throwing an internal NullPointerException and failing out because one of those values came back null. You could possibly solve this with the following:
List<string> outputRoleUserLIst2=roleUserList
.Where(x => x != null && x.UserID != null && x.RoleID != null && missingRoleUserList
.Where(y => y != null && y.UserID != null && y.RoleID != null && y.RoleID!=x.RoleID && y.UserID!=x.UserID)
.FirstOrDefault()!=null)
.Select(x => x.UserID + ",\"" + x.RoleID).Distinct().ToList();
This is not pretty, and this is the other Linq syntax (with which I am more comfortable) but hopefully you understand what I am going for here. I'd be curious to know what would happen if you dropped this into your program (If I've guessed all of your meanings correctly!). I'll look back in a bit to see if you have added any information!