We Keep Coding

How to convert SQL with LEFT JOIN to EF CORE 3 LINQ

I need to convert this SQL:
SELECT
sb.Id AS id,
sb.name as name,
sb.age AS age,
sb.BirthDay as BirthDay,
su.UnitId AS unitId,
sb.WorkedInDodo AS workedInDodo,
sb.RelativesWorkedInDodo AS relativesWorkedInDodo,
sb.PhoneNumber AS phoneNumber,
sb.Email AS email,
sb.DeliveryCheck AS deliveryCheck,
sb.RestaurantCheck AS restaurantCheck,
sb.StandardsRatingCheck as standardsRatingCheck,
sb.SocialNetwork AS socialNetwork,
sb.SocialNetworkId AS socialNetworkId,
sb.socialNetworkScreenName AS socialNetworkScreenName,
sb.SourceOfInformationAboutSecretBuyers AS SourceOfInformationAboutSecretBuyers,
sb.suitable AS suitable,
sb.SocialNetworkMessagingEnable as socialNetworkMessagingEnable,
sb.IsInGroup as isInGroup,
sb.IsKeyWord as IsKeyWord,
sb.IsBanned as IsBanned,
sb.IsFraud as IsFraud,
sb.LastUnitsUpdateUtcDate as lastUnitsUpdateUtcDate,
sb.Comments as comments,
sb.MessagingApproval as messagingApproval,
sb.ProcessDataApproval as processDataApproval,
sb.CreatedDateTimeUtc AS createdDateTimeUtc,
sb.ModifiedDateTineUTC AS modifiedDateTime,
sb.Country AS country,
sb.PhoneNumberEditedUtc AS PhoneNumberEditedUtc
FROM (select * from
(SELECT s.*,
IF(s.LastSearchMessageDateUtc is null, 0, 1) as searched,
IF(cc.Id is null, 0, 1) as onCheck
FROM secretbuyers s
JOIN secretbuyerunits sbu ON sbu.SecretBuyerId = s.Id
LEFT JOIN outgoingMessageQueue mq ON mq.Type = 1 AND mq.VkUserId = s.SocialNetworkId and mq.State in (1,2)
LEFT JOIN checkcandidates cc ON cc.SecretBuyerId = s.id AND cc.State IN (1,4) AND cc.Date >= #p_checkBeginDateTime AND cc.Date <= #p_checkEndDateTime
WHERE sbu.UnitId = #p_unitId
AND (s.LastSearchMessageDateUtc is null OR s.LastSearchMessageDateUtc < #p_lastSearchMessageDateUtcLimit)
AND ((#p_deliveryCheck = true AND s.DeliveryCheck = true) OR (#p_restaurantCheck = true AND s.RestaurantCheck = true) OR (#p_standardsRatingCheck = true AND s.StandardsRatingCheck = true))
AND s.Suitable = true
AND s.IsRemove = false
AND mq.Id is null
AND s.IsBanned = false
AND s.IsFraud = false
AND s.IsAutoSearchable = true
group by s.Id
Order By onCheck ASC, searched asc, s.LastSearchMessageDateUtc asc) as temp
LIMIT #p_count) AS sb
JOIN secretbuyerunits su ON su.SecretBuyerId = sb.Id;
to EF Core 3.0 linq. The main problem is left joins. At first I've tried to do it like this:
private async Task<IEnumerable<MysteryShopper>> FindMysteryShoppers(
SearchMysteryShoppersParameters searchParameters, CancellationToken ct)
{
var lastSearchEdge = _nowProvider.UtcNow().Date.AddDays(-3);
bool shouldHaveDeliveryCheckMark = ShouldHaveDeliveryCheckMark(searchParameters.SearchType);
bool shouldHaveRestaurantCheckMark = ShouldHaveRestaurantCheckMark(searchParameters.SearchType);
bool shouldHaveStandardsCheckMark = ShouldHaveStandardsCheckMark(searchParameters.SearchType);
var lastCheckupEdgeDate = searchParameters.CheckupDates.Max().AddDays(-30);
return await _ratingsContext.SecretBuyers.AsNoTracking().Where(ms => !ms.IsBanned &&
!ms.IsFraud
&& ms.IsAutoSearchable
&& ms.Suitable
&& !ms.IsRemove
&& ms.SocialNetworkMessagingEnable
&& (ms.LastSearchMessageDateUtc == null
|| ms.LastSearchMessageDateUtc < lastSearchEdge)
)
.Where(ms => !shouldHaveDeliveryCheckMark && ms.DeliveryCheck)
.Where(ms => !shouldHaveRestaurantCheckMark && ms.RestaurantCheck)
.Where(ms => !shouldHaveStandardsCheckMark && ms.StandardsRatingCheck)
.Where(ms => ms.MysteryShopperUnits
.Any(u => searchParameters.UnitIds.Contains(u.UnitId))
)
.GroupJoin(_ratingsContext.Checkups.AsNoTracking().Where(cc => !_cancelledStates.Contains(cc.State)), ms => ms.Id,
cc => cc.SecretBuyerId, (ms, cc) => new
{
MysteryShopper = ms,
LastCheckupDate = cc
.Max(c => c.Date)
}
)
.GroupJoin(_ratingsContext.Messages.AsNoTracking().Where(m =>
m.Type == OutgoingMessageType.CandidateSearch
&& (m.State == OutgoingMessageState.Sending || m.State == OutgoingMessageState.Waiting)),
ms => ms.MysteryShopper.SocialNetworkId,
m => m.VkUserId,
(ms, m) =>
new {ms.MysteryShopper, ms.LastCheckupDate, HasPendingMessages = m.Any()})
.Where(ms => !ms.HasPendingMessages)
.Where(ms => ms.LastCheckupDate < lastCheckupEdgeDate)
.OrderBy(ms => ms.LastCheckupDate != null)
.ThenBy(ms => ms.MysteryShopper.LastSearchMessageDateUtc != null)
.ThenBy(ms => ms.MysteryShopper.LastSearchMessageDateUtc)
.Select(ms => ms.MysteryShopper)
.ToArrayAsync(ct);
}
After running this query I got error:
... by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.
After that I discovered from this question that you can't make group by queries evaluated on client anymore.
After more searching I've discovered another solution. So i've re-written my method like this:
private async Task<IEnumerable<MysteryShopper>> FindMysteryShoppers(
SearchMysteryShoppersParameters searchParameters, CancellationToken ct)
{
var lastSearchEdge = _nowProvider.UtcNow().Date.AddDays(-3);
bool shouldHaveDeliveryCheckMark = ShouldHaveDeliveryCheckMark(searchParameters.SearchType);
bool shouldHaveRestaurantCheckMark = ShouldHaveRestaurantCheckMark(searchParameters.SearchType);
bool shouldHaveStandardsCheckMark = ShouldHaveStandardsCheckMark(searchParameters.SearchType);
var lastCheckupEdgeDate = searchParameters.CheckupDates.Max().AddDays(-30);
var mysteryShoppersLeftJoinedWithCheckups = from ms in _ratingsContext.Set<MysteryShopper>()
where ms.MysteryShopperUnits
.Any(u => searchParameters.UnitIds.Contains(u.UnitId)
&& !shouldHaveStandardsCheckMark && ms.StandardsRatingCheck
&& !shouldHaveRestaurantCheckMark && ms.RestaurantCheck
&& !shouldHaveDeliveryCheckMark && ms.DeliveryCheck
&& !ms.IsBanned
&& !ms.IsFraud
&& ms.IsAutoSearchable
&& ms.Suitable
&& !ms.IsRemove
&& ms.SocialNetworkMessagingEnable
&& (ms.LastSearchMessageDateUtc == null || ms.LastSearchMessageDateUtc < lastSearchEdge))
join cc in _ratingsContext.Set<Checkup>()
on ms.Id equals cc.SecretBuyerId into checkups
from cc in checkups.DefaultIfEmpty()
where !_cancelledStates.Contains(cc.State)
select new {MysteryShopper = ms, LastCheckupDate = checkups.Max(c => c.Date)};
var mysteryShoppersLeftJoinedWithCheckupsAndMessages =
from mysteryShopperWithCheckup in mysteryShoppersLeftJoinedWithCheckups
join m in _ratingsContext.Set<Message>()
on mysteryShopperWithCheckup.MysteryShopper.SocialNetworkId equals m.VkUserId into messages
from m in messages.DefaultIfEmpty()
where m.Type == OutgoingMessageType.CandidateSearch && (m.State == OutgoingMessageState.Sending ||
m.State == OutgoingMessageState.Waiting)
select new
{
mysteryShopperWithCheckup.MysteryShopper,
mysteryShopperWithCheckup.LastCheckupDate,
HasPendingMessages = messages.Any()
};
var orderedMysteryShoppersWithoutPendingMessage =
from mysteryShopperWithCheckupAndMessage in mysteryShoppersLeftJoinedWithCheckupsAndMessages
where !mysteryShopperWithCheckupAndMessage.HasPendingMessages &&
mysteryShopperWithCheckupAndMessage.LastCheckupDate < lastCheckupEdgeDate
orderby mysteryShopperWithCheckupAndMessage.LastCheckupDate != null,
mysteryShopperWithCheckupAndMessage.MysteryShopper.LastSearchMessageDateUtc != null,
mysteryShopperWithCheckupAndMessage.MysteryShopper.LastSearchMessageDateUtc
select mysteryShopperWithCheckupAndMessage.MysteryShopper;
return await orderedMysteryShoppersWithoutPendingMessage.ToArrayAsync(ct);
}
But now i get another error:
could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().
After some more testing i know for sure that checkups.Max(c => c.Date)} and HasPendingMessages = messages.Any() is the source of this error. How to fix this query? You can suggest completely different solution or approach. The main idea is to bring logic from sql to c#.
PS: Sorry for long question.

Added navigation properties and nugget Z.EntityFramework.Plus (for IncludeFilter) and rewritten query like this:
var query = _ratingsContext.SecretBuyers
.AsNoTracking()
.IncludeFilter(ms => ms.Checkups.Where(cc => !_cancelledStates.Contains(cc.State)))
.IncludeFilter(ms => ms.Messages.Where(m => m.Type == OutgoingMessageType.CandidateSearch &&
(m.State == OutgoingMessageState.Sending ||
m.State == OutgoingMessageState.Waiting)))
.Where(ms => !ms.IsBanned
&& !ms.IsFraud
&& ms.IsAutoSearchable
&& ms.Suitable
&& !ms.IsRemove
&& ms.SocialNetworkMessagingEnable
&& (ms.LastSearchMessageDateUtc == null || ms.LastSearchMessageDateUtc < lastSearchEdge)
)
.Where(ms => !shouldHaveDeliveryCheckMark || ms.DeliveryCheck)
.Where(ms => !shouldHaveRestaurantCheckMark || ms.RestaurantCheck)
.Where(ms => !shouldHaveStandardsCheckMark || ms.StandardsRatingCheck)
.Where(ms => ms.MysteryShopperUnits
.Any(u => searchParameters.UnitIds.Contains(u.UnitId))
)
.Where(ms => !ms.Messages.Any())
.Select(x => new {MysteryShopper = x, LastCheckupDate = x.Checkups.Max(c => c.Date)})
.Where(ms => ms.LastCheckupDate < lastCheckupEdgeDate)
.OrderBy(x => x.LastCheckupDate != null)
.ThenBy(x => x.MysteryShopper.LastSearchMessageDateUtc != null)
.ThenBy(x => x.MysteryShopper.LastSearchMessageDateUtc)
.Select(x => x.MysteryShopper)
.Take(searchParameters.MessagesPerUnit);

Difference between LINQ Lambda and SQL statement

I have the following lambda statement:
var resources = Db.Resource.Where(w => w.ResValue.Any(a => a.ApplicationFk == applicationPk) && w.CategoryFk == (categoryId ?? w.CategoryFk ) && w.IsEditable);
if (cultureIdsMissing!= null)
{
resources = resources.Where(w => w.ResValue.Any(a => cultureIdsMissing.Any(aa => aa == a.CultureFk) && a.Value == string.Empty));
}
This is not returning the result which I want, which is returned by:
SELECT Resource.ResourcePk, Resource.CategoryFk, Resource.Name, Resource.IsEditable, ResValue.ApplicatieFk, ResValue.CultureFk, ResValue.Value
FROM Resource
INNER JOIN ResValue ON Resource.ResourcePk = ResValue.ResourceFk
WHERE (ResValue.ApplicatieFk = 6)
AND (Resource.IsEditable = 1)
AND (ResValue.Value = '')
AND (ResValue.CultureFk = 1 OR ResValue.CultureFk = 2)
Not that cultureIdsMissing is a List containing both the numbers 1 and 2.
What am I missing or doing wrong with the lambda query?

I think you have to remove && w.CategoryFk == (categoryId ?? w.CategoryFk ) from your linq lemda expression. if categoryId = 1 then it will take only records with value 1. So try after remove that. Your linq code should be this.
var resources = Db.Resource.Where(w => w.ResValue.Any(a => a.ApplicationFk == applicationPk)&& w.IsEditable);
if (cultureIdsMissing!= null)
{
resources = resources.Where(w => w.ResValue.Any(a => cultureIdsMissing.Any(aa => aa == a.CultureFk) && a.Value == string.Empty));
}

You should take it from your sql statement :
Db.Resource
.Join(Db.ResValue
, rs => rs.ResourcePk
, resV => resv.resourceFk
, (rs, resv) => new { res = rs, resV = resV })
.Where(w => w.resv.ApplicatieFk == 6
&& w.res ==1
&& resv.Value == string.empty()
&& (resv.CultureFk == 1 || resv.CultureFk == 2))
It's not tested so maybe it won't work on first try.

I would translate the SQL to query comprehension syntax. In general, convert phrases in query comprehension order, use table aliases as range variables (or create range variables), and put unary/overall aggregate functions (such as TOP, DISTINCT or SUM) as function calls outside the whole query. For your SQL,
var ans = from r in Resource
where r.IsEditable == 1
join rv in ResValue on r.ResourcePk equals rv.ResourceFk
where rv.ApplicatieFk == 6 && rv.Value == "" && (rv.CultureFk == 1 || rv.CultureFk == 2)
select new { r.ResourcePk, r.CategoryFk, r.Name, r.IsEditable, rv.ApplicatieFk, rv.CultureFk, rv.Value };

Summing a column gives null in C#/ASP.NET

I have the following sql statement that calculate the sum of the column:
select coalesce(SUM(cdin_ActMortgageAmnt),0)
from CDIndex,company
where comp_companyid=cdin_companyid and comp_idcust like '%10319%'
and cdin_Deleted is null and cdin_startunstufdate is not null
and cdin_Status='InProgress'
gives me the output like this:
I tried to convert it to LINQ like this:
var sumation = (from com in db.Companies
join cd in db.CDIndexes on com.Comp_CompanyId equals cd.cdin_CompanyId
where
cd.cdin_Status == "InProgress" &&
cd.cdin_startunstufdate == null &&
cd.cdin_Deleted == null
select new {
sum = cd.cdin_ActMortgageAmnt
}
);
var summ = sumation.Sum(x => x.sum);
When I put tracePoint beside var summ in debug mode it gives me null when i point to it.
What is the problem?

On your case you are using coalesce(SUM(cdin_ActMortgageAmnt),0) because some values of cdin_ActMortgageAmnt can be null and you are giving the default value of 0, you need to do the same in your final query. Something like this when you do the select
cd.cdin_ActMortgageAmnt ?? 0

this query is appropriate your sql query
var sumation =db.Companies.Join(db.CDIndexes,
com=>com.Comp_CompanyId,
cd=>cd.cdin_companyid,
(com,cd)=>new {com,cd})
.Where(x=>x.com.comp_idcust.Contains("10319") && x.cd.cdin_Status== "InProgress" &&
cd.cdin_startunstufdate != null &&
cd.cdin_Deleted == null)
.Select(x=>new
{
sum=x.cd.cdin_ActMortgageAmnt ?? 0
}).ToList()

Here is one way:
var summ = db.Companies.Join(
db.CDIndexes,
cd => cd.cdin_CompanyId,
com => Comp_CompanyId,
(com, cd) => new { com, cd })
.Where(z=>z.com.comp_idcust.Contains("10319")) // Added "LIKE"
.Where(z=>z.cd.cdin_Status == "InProgress")
.Where(z=>z.cd.cdin_startunstufdate != null) // Reversed your condition
.Where(z=>z.cd.cdin_Deleted == null)
.Sum(z=>z.cd.cdin_ActMortgageAmnt);
You can also combine all the Wheres together, but I prefer not to in most cases like this:
var summ = db.Companies.Join(
db.CDIndexes,
cd => cd.cdin_CompanyId,
com => Comp_CompanyId,
(com, cd) => new { com, cd })
.Where(z=>z.com.comp_idcust.Contains("10319") // Added "LIKE"
&& z.cd.cdin_Status == "InProgress"
&& z.cd.cdin_startunstufdate != null // Reversed your condition
&& z.cd.cdin_Deleted == null)
.Sum(z=>z.cd.cdin_ActMortgageAmnt);

Dynamically Building LINQ-To-Entities Where Clause

How can I build the where clause dynamically, Some time the OwnerID will be zero only itemID and LocationIDwill be provided as the search criteria, in that case the LINQ should be
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.ItemId == searchCriteria.ItemID &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Some time the OwnerID and ItemId will be zero then only the LocationID will be provided as the search criteria, in that case the LINQ should be
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Some time the whole OwnerID, ItemID and LocationID will be provided as the search criteria, then the LINQ will be like this
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.OwnerId == searchCriteria.OwnerID &&
s.ItemId == searchCriteria.ItemID &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Here only the where clause is changing, Please help me to solve. How I can I build the where clause dynamically (Please note, here I am having a navigation property which is OwnerDetails.LocationId).

You can easily do it by using method-based query. You can add the conditions one at a time and call Select and ToList at the end:
// Where(x => true) might not be necessary, you can try skipping it.
var query = repository.ItemOwners.Include("OwnerDetails").Where(x => true);
if (searchCriteria.OwnerID != null)
query = query.Where(s => s.OwnerID == searchCriteria.OwnerID);
if (searchCriteria.ItemID != null)
query = query.Where(s => s.ItemID == searchCriteria.ItemID);
if (searchCriteria.OwnerID != null)
query = query.Where(s => s..OwnerDetails.LocationId == searchCriteria.LocationID);
var results = query.Select(s => new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();

Simplest is just check the zero condition in the Where clause:
(from s in repository.ItemOwners.Include("OwnerDetails")
where (searchCriteria.OwnerID == 0 || s.OwnerId == searchCriteria.OwnerID) &&
(searchCriteria.ItemID == 0 || s.ItemId == searchCriteria.ItemID) &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();

Select two columns from the table via linq

I use the query below to get all columns(20 more) in Entity Framework Linq. Because of out of memory exception, I only want to get two of them. One is "FileName", the other one is "FilePath". How to modify my code?
var query = DBContext.Table1
.Where(c => c.FacilityID == facilityID && c.FilePath != null && c.TimeStationOffHook < oldDate)
.OrderBy(c => c.FilePath)
.Skip(1000)
.Take(1000)
.ToList();
foreach(var t in query)
{
Console.WriteLine(t.FilePath +"\\"+t.FileName);
}

var query = DBContext.Table1.Where(c => c.FacilityID == facilityID && c.FilePath != null && c.TimeStationOffHook < oldDate)
.OrderBy(c => c.FilePath)
.Skip(1000)
.Take(1000)
.Select(c => new { c.FilePath, c.FileName })
.ToList();
foreach(var t in query)
{
Console.WriteLine(t.FilePath +"\\"+t.FileName);
}
You need to use Select.

Just select out two of the columns:
DBContext.Table1.Select(c => new { c.FileName, c.FilePath });

How about something like
using (var entity = new MyModel(ConnectionString))
{
var query = (from myTable in entity.theTable
where myTable.FacilityID == facilityID &&
myTable.FilePath != null &&
myTable.TimeStationOffHook < oldDate
orderby myTable.FilePath
select new
{
myTable,FileName,
myTable.FilePath
}).Skip(1000).Take(1000).ToList();
//do what you want with the query result here
}