We Keep Coding

C# Entity Framework Pagination Children Includes

I am looking for the fastest way to do the following operation. What I need to accomplish, is I have a screen that displays the "Parts" that are defined inside of a "Lot". Each part has objects of a station, and each station has objects of tools, and each tool can have measurements.
My problem is I cannot get the pagination to work. The incoming offset is 0, and the number of records to take is 20 however the following operation is not working:
Lot foundLot = EntitiesContext.Lots.Where(x => x.ID == lotID)
.IncludeFilter(lot => lot.Parts.OrderBy(n => n.PartID).Skip(offset).Take(numberOfRecords).ToList())
.IncludeOptimized(lot => lot.Parts.Select(parts => parts.Stations))
.IncludeOptimized(lot => lot.Parts.Select(parts => parts.Stations.Select(station => station.Tools)))
.IncludeOptimized(lot => lot.Parts.Select(parts => parts.Stations.Select(station =>
station.Tools.Select(tools => tools.Measurements))))
.FirstOrDefault();
So I am trying to filter to only grab certain parts, and then of those filtered parts, I want to grab all of the children's data associated with them. I have checked all of the existing stack overflow articles related to this, and the changes I make either result in the Z.EntityFrameworkPlus package throwing a generic exception, that provides no details (which is what the above code does), or if I use the regular EntityFramework functions it throws an exception for the invalid path.
Thank you for your assistance.

Okay, after reading the docs for Z.EntityFrameworkPlus I now see you cannot use IncludeFilter and IncludeOptimzed on the same LINQ statement
https://entityframework-plus.net/query-include-optimized
I switched IncludeFilter to IncludeOptimzed and it works.

Orderby C# string record

I have the following orderby for a record read from db and then building a string.
The following code works fine but I know this can be improved any suggestion is highly appreciated.
result.Sites.ForEach(x =>
{
result.SiteDetails +=
string.Concat(ICMSRepository.Instance.GetSiteInformationById(x.SiteInformationId).SiteCode,
",");
});
//Sort(Orderby) sites by string value NOT by numerical order
result.SiteDetails = result.SiteDetails.Trim(',');
List<string> siteCodes = result.SiteDetails.Split(',').ToList();
var siteCodesOrder = siteCodes.OrderBy(x => x).ToArray();
string siteCodesSorted = string.Join(", ", siteCodesOrder);
result.SiteDetails = siteCodesSorted;

That's a little convoluted, yeah.
All we need to do is select out the SiteCode as string, sort with OrderBy, then join the results. Since String::Join has a variant that works with IEnumerable<string> we don't need to convert to array in the middle.
What we end up with is a single statement for assigning to your SiteDetails member:
result.SiteDetails = string.Join(", ",
result.Sites
.Select(x => $"{ICMSRepository.Instance.GetSiteInformationById(x.SiteInformationId).SiteCode}")
.OrderBy(x => x)
);
(Or you could use .ToString() instead of $"{...}")
This is the general process for most transforms in LINQ. Figure out what your inputs are, what you need to do with them, and how the outputs should look.
If you're using LINQ it's uncommon that you will have to build and manipulate intermediary lists unless you're doing something quite complex. For simple tasks like sorting a sequence of values there is almost never a reason to put them into transitional collections, since the framework handles all of that for you.
And the best part is it enumerates the collection one time to get the full set of data. No more loops to pull the data out, then process, then rebuild.

One thing that will improve performance is to get rid of the .ToList() and the .ToString. Neither is necessary and just take up extra processing time and memory.

Go with Corey's answer, which this is a variant of, but I thought I'd offer a slightly clearer way to express the query:
result.SiteDetails =
String.Join(", ",
from x in result.Sites
let sc = ICMSRepository.Instance.GetSiteInformationById(x.SiteInformationId).SiteCode
orderby sc
select sc);

Linq performance when diffing two lists using inner Contains

EDIT 01: I seem to have found a solution (click for the answer) that works for me. Going from and hour to merely seconds by pre-computing and then applying the .Except() extension method; but leaving this open if anyone else encounters this problem or if anyone else finds a better solution.
ORIGINAL QUESTION
I have the following set of queries, for differend kind of objects I'm staging from a source system so I can keep it in sync and make a delta stamp myself, as the sourcesystem doesn't provide it, nor can we build or touch it.
I get all data in memory an then for example perform this query, where I look for objects that don't exist any longer in the source system, but are present in the staging database - and thus have to be marked "deleted". The bottleneck is the first part of the LINQ query - on the .Contains(), how can I improve it's performance - mayve with .Except(), with a custom comparer?
Or should I best put them in a hashing list and them perform the compare?
The problem is though I have to have the staged objects afterwards to do some property transforms on them, this seemed the simplest solution, but unfortunately it's very slow on 20k objects
stagedSystemObjects.Where(stagedSystemObject =>
!sourceSystemObjects.Select(sourceSystemObject => sourceSystemObject.Code)
.Contains(stagedSystemObject.Code)
)
.Select(x =>
{
x.ActiveStatus = ActiveStatuses.Disabled;
x.ChangeReason = ChangeReasons.Edited;
return x;
})
.ToList();

Based on Yves Schelpe's answer. I made a little tweaks to make it faster.
The basic idea is to cancel the first two ToList and use PLINQ. See if this help
var stagedSystemCodes = stagedSystemObjects.Select(x => x.Code);
var sourceSystemCodes = sourceSystemObjects.Select(x => x.Code);
var codesThatNoLongerExistInSourceSystem = stagedSystemCodes.Except(sourceSystemCodes).ToArray();
var y = stagedSystemObjects.AsParallel()
.Where(stagedSystemObject =>
codesThatNoLongerExistInSourceSystem.Contains(stagedSystemObject.Code))
.Select(x =>
{
x.ActiveStatus = ActiveStatuses.Disabled;
x.ChangeReason = ChangeReasons.Edited;
return x;
}).ToArray();
Note that PLINQ may only work well for computational limited task with multi-core CPU. It could make things worse in other scenarios.

I have found a solution for this problem - which brought it down to mere seconds in stead of an hour for 200k objects.
It's done by pre-computing and then applying the .Except() extension method
So no longer "chaining" linq queries, or doing .Contains inside a method... but make it "simpler" by first projecting both to a list of strings, so that inner calculation doesn't have to happen over and over again in the original question's example code.
Here is my solution, that for now is satisfactory. However I'm leaving this open if anyone comes up with a refined/better solution!
var stagedSystemCodes = stagedSystemObjects.Select(x => x.Code).ToList();
var sourceSystemCodes = sourceSystemObjects.Select(x => x.Code).ToList();
var codesThatNoLongerExistInSourceSystem = stagedSystemCodes.Except(sourceSystemCodes).ToList();
return stagedSystemObjects
.Where(stagedSystemObject =>
codesThatNoLongerExistInSourceSystem.Contains(stagedSystemObject.Code))
.Select(x =>
{
x.ActiveStatus = ActiveStatuses.Disabled;
x.ChangeReason = ChangeReasons.Edited;
return x;
})
.ToList();

Linq Orderby 'string contains' not behaving as expected

I'm having a strange issue trying to order a list based on whether or not it contains a particular string value. Within a view, I have a list on my viewmodel for which I would like to display those objects which are not "snoozed" first and then ordered by whether or not their title contains the word "order". So my code is as follows:
var sortedFaults = accountFaultContainer.Faults.Values.OrderBy(f => f.IsSnoozed).ThenByDescending(f => f.Title.ToLower().Contains("order"));
I then loop through this list and add the 'faults' to my table using a foreach loop...
foreach (Fault fault in sortedFaults)
This works in some cases but not in others. I've been trying to single out those that do not behave as expected but there is no clear difference. For example, in once case, a title of "Full failing sync" was ordered both before and after "Failing order syncs". But it is not the case that this always misbehaves; in some tables the ordering is working.
Has anybody come across a similar issue before? Or is there something blatantly obvious I'm missing? I'm scratching my head here!

You order by IsSnoozed first, so that takes priority.
.OrderBy(f => f.IsSnoozed)
.ThenByDescending(f => f.Title.ToLower().Contains("order"));
Try it the other way around:
.OrderBy(f => f.Title.ToLower().Contains("order"))
.ThenByDescending(f => f.IsSnoozed);

Searching Multiple Fields with LINQ Contains or Other

I'm using LINQ to search multiple fields on a single phrase and I'm using Contains() to do this. Up until today, when I noticed that the find method isn't working correctly.
So I'd like to be able to search each of the fields where there is any match. I've done some searching on Google and through SO and found suggestions of using Any()? Does anyone have any suggestions?
var l = _getData().Select(_marshallData).ToList();
return l.Where(x => x.Name.Contains(what) || x.Pumpkin.Contains(what) || x.Orange.Contains(what) || x.Tomato.Contains(what)).ToList();
Please excuse the stupid field names, I've had to change them for confidentiality.

Since you're materializing the data already you could do this:
var l = _getData().Select(_marshallData).AsEnumerable();
return l.Where(x => new[] { x.Name, x.Pumpkin, x.Orange, x.Tomato }.Any(s => s.Contains(what)));
But if you're using an ORM (like Entity Framework) this trick probably won't work (since there's a good chance this can't be converted to a SQL expression). In this case, you're original solution is fine, but it would probably be better to do this before you materialize the data.
return _getData()
.Where(x =>
x.Name.Contains(what) || x.Pumpkin.Contains(what) ||
x.Orange.Contains(what) || x.Tomato.Contains(what))
.Select(_marshallData)
.AsEnumerable();
Of course the field names here might be different, since the parameter x is probably a different type. Without further information, it's up to you to write this filter correctly.
Note that in both examples, I've eliminated the calls to ToList. Unless you absolutely need the result to a List<T> this is probably unnecessary.