We Keep Coding

Comparing 2 Csv files having 12 million record in each file by sorting them first gives Out Of Memory Exception

I am trying to compare two Csv files that has 12 million records in each file, Before comparing files I want to sort them in ascending order and then compare them. I have tried below code but it gives Out Of Memory Exception.
public void SortCsv(string originalPath1, string originalPath2)
{
var lines1 = File.ReadLines(originalPath1, Encoding.Default);
var sorted1 = lines1.Skip(1).OrderBy(row => row).Select(row => row).ToList();
var lines2 = File.ReadLines(originalPath2, Encoding.Default);
var sorted2 = lines2.Skip(1).OrderBy(row => row).Select(row => row).ToList();
var extraFiles1Csv = sorted1.Except(sorted2, StringComparer.CurrentCultureIgnoreCase);
var extraFiles2Csv = sorted2.Except(sorted1, StringComparer.CurrentCultureIgnoreCase);
if(extraFiles1Csv.Count() > 0)
foreach (var item in extraFiles1Csv)
Console.WriteLine(item);
if (extraFiles1Csv.Count() > 0)
foreach (var item in extraFiles2Csv)
Console.WriteLine(item);
Console.WriteLine("Record is matched");
Console.ReadKey();
}
I am expecting the that it should sort the data of both csv file and compare them and if there is any mismatch then it should return mismatch record else it should print files matched.

By having linesX and sortedX like that you are keeping each file in memory 2ce
and i think you can drop the select
combining it would halve the memory usage
var lines1 = File.ReadLines(originalPath1, Encoding.Default).Skip(1).OrderBy(row => row).ToList();
further i would have a look at some csv nuget package like Sylvan.Data.Csv
to read your files, these typically have better speed and memory handeling than trying to do it yourself. (and object mapping 2 ;) )

foreach in foreach list c#

The GalleryDetail.Id is 148 when it enters the first foreach loop, and the GalleryDetail.Id is 148 when it enters the second foreach loop. But it does not enter the first foreach loop again. It continues from the second. How do I get it to re-enter the first loop here?
NOTE: I do not have direct access from the GalleryDetail.Id request.
var detailList = await _repo.GalleryDetail.GetAllAsync();
foreach (var item in mapped.GalleryDetails)
{
foreach (var item2 in detailList)
{
if (item.Id != item2.Id)
{
var mapped3 = _mapper.Map<GalleryDetails>(item2);
await _repo.GalleryDetail.DeleteAsync(mapped3);
}
}
}

It's not necessary to use 2 loop here, you can use LinQ instead. And you should not leave async Delete inside foreach loop, because it will connect to you database mutiple time. Example:
var detailList = await _repo.GalleryDetail.GetAllAsync();
//Map the whole list
var detailListMapped = _mapper.Map<List<GalleryDetails>>(detailList);
//Use LinQ to find database items not in request
var deletedList = detailListMapped.Where(x => !mapped.GalleryDetails.Any(y => y.Id == x.Id)).ToList();
//Entity Framework have RemoveRange function, you should use it here
await _repo.GalleryDetail.RemoveRangeAsync(deletedList );

Loop to check for duplicate strings

I want to create a loop to check a list of titles for duplicates.
I currently have this:
var productTitles = SeleniumContext.Driver.FindElements(By.XPath(ComparisonTableElements.ProductTitle));
foreach (var x in productTitles)
{
var title = x.Text;
productTitles = SeleniumContext.Driver.FindElements(By.XPath(ComparisonTableElements.ProductTitle));
foreach (var y in productTitles.Skip(productTitles.IndexOf(x) + 1))
{
if (title == y.Text)
{
Assert.Fail("Found duplicate product in the table");
}
}
}
But this is taken the item I skip out of the array for the next loop so item 2 never checks it's the same as item 1, it moves straight to item 3.
I was under the impression that skip just passed over the index you pass in rather than removing it from the list.

You can use GroupBy:
var anyDuplicates = SeleniumContext
.Driver
.FindElements(By.XPath(ComparisonTableElements.ProductTitle))
.GroupBy(p => p.Text, p => p)
.Any(g => g.Count() > 1);
Assert.That(anyDuplicates, Is.False);
or Distinct:
var productTitles = SeleniumContext
.Driver
.FindElements(By.XPath(ComparisonTableElements.ProductTitle))
.Select(p => p.Text)
.ToArray();
var distinctProductTitles = productTitles.Distinct().ToArray();
Assert.AreEqual(productTitles.Length, distinctProductTitles.Length);
Or, if it is enough to find a first duplicate without counting all of them it's better to use a HashSet<T>:
var titles = new HashSet<string>();
foreach (var title in SeleniumContext
.Driver
.FindElements(By.XPath(ComparisonTableElements.ProductTitle))
.Select(p => p.Text))
{
if (!titles.Add(title))
{
Assert.Fail("Found duplicate product in the table");
}
}
All approaches are better in terms of computational complexity (O(n)) than what you propose (O(n2)).

You don't need a loop. Simply use the Where() function to find all same titles, and if there is more than one, then they're duplicates:
var productTitles = SeleniumContext.Driver.FindElements(By.XPath(ComparisonTableElements.ProductTitle));
foreach(var x in productTitles) {
if (productTitles.Where(y => x.Text == y.Text).Count() > 1) {
Assert.Fail("Found duplicate product in the table");
}
}

I would try a slightly different way since you only need to check for duplicates in a one-dimensional array.
You only have to check the previous element with the next element within the array/collection so using Linq to iterate through all of the items seems a bit unnecessary.
Here's a piece of code to better understand:
var productTitles = SeleniumContext.Driver.FindElements(By.XPath(ComparisonTableElements.ProductTitle))
for ( int i = 0; i < productionTitles.Length; i++ )
{
var currentObject = productionTitles[i];
for ( int j = i + 1; j < productionTitles.Length; j++ )
{
if ( currentObject.Title == productionTitles[j].Title )
{
// here's your duplicate
}
}
}
Since you've checked that item at index 0 is not the same as item placed at index 3 there's no need to check that again when you're at index 3. The items will remain the same.

The Skip(IEnumerable, n) method returns an IEnumerable that doesn't "contain" the n first element of the IEnumerable it's called on.
Also I don't know what sort of behaviour could arise from this, but I wouldn't assign a new IEnumerable to the variable over which the foreach is being executed.
Here's another possible solution with LINQ:
int i = 0;
foreach (var x in productTitles)
{
var possibleDuplicate = productTitles.Skip(i++).Find((y) => y.title == x.title);
//if possibleDuplicate is not default value of type
//do stuff here
}
This goes without saying, but the best solution for you will depend on what you are trying to do. Also, I think the Skip method call is more trouble than it's worth, as I'm pretty sure it will most certainly make the search less eficient.

How to get next item from list based on some condition in C#?

I have a List which contains set of time frames (DataTime format). It has StartDateTime & EndDateTime. I am trying to get a next item of the list based on a condition. How can I do that?
For Example,
foreach (var currentTimeSlot in prepBlock.EligiblePickupTimes.BlockList)
{
if (potentialStartTime > currentTimeSlot.EndDateTime)
{
//Skip current time slot and grab next one and so on.
}
}

You can use FirstOrDefault in order to get the first item matching your predicate:
prepBlock.EligiblePickupTimes.BlockList
.FirstOrDefault(x => potentialStartTime <= x.EndDateTime);
You can get the entire Enumerable<T> of items from the first matches this condition to the end using SkipWhile:
prepBlock.EligiblePickupTimes.BlockList
.SkipWhile(x => potentialStartTime > x.EndDateTime);
The first condition is equivalent to the following code:
prepBlock.EligiblePickupTimes.BlockList
.SkipWhile(x => potentialStartTime > x.EndDateTime)
.FirstOrDefault();
From what is see you try to do in the image you can do the following:
returnValue.IsEstimateSuccessful &= !prepBlock.EligiblePickupTimes.BlockList
.SkipWhile(x => potentialStartTime > x.EndDateTime)
.Any();

Unless I'm missing something, I believe you can accomplish this with the .FirstOrDefault method as mentioned in comments.
using System.Linq;
...
var nextAvailableItem =
prepBlock.EligiblePickupTimes.BlockList
// reversing your condition above to find value I want
// instead of specifying values I don't want
.FirstOrDefault(x => potentialStartTime <= x.EndDateTime)
;
// did we find a value to match our condition?
var wasFound = nextAvailableItem != default(DateTime);

If you are just trying to loop through all the timeslots where potentialStartTime is greater than it's EndDateTime, then:
foreach (var currentTimeSlot in
prepBlock.EligiblePickupTimes.BlockList.Where(x=>potentialStartTime > x.EndDateTime))
{
}
based on your image, I think this is what you are looking for however:
returnValue.IsEstimateSuccessful=!prepBlock
.EligiblePickupTimes
.BlockList
.Any(x=>potentialStartTime > x.EndDateTime);
if returnValue.IsEstimateSuccessful is set prior to that (like if you default it to true, and many checks might turn it false):
returnValue.IsEstimateSuccessful&=!prepBlock
.EligiblePickupTimes
.BlockList
.Any(x=>potentialStartTime > x.EndDateTime);

LINQ for removing elements that are started with other element from list

I have a list List<string> with some paths.
C:\Dir\Test\
C:\MyDir\
C:\YourDir\
C:\Dir\
I want to go through all the elements (using LINQ) and remove entries that are started with other element from my list.
In my example C:\Dir\Test\ starts with C:\Dir\ - so I want to remove C:\Dir\Test\.

Use List<T>.RemoveAll() method:
sourceList.RemoveAll(x => sourceList.Any(y => x != y && x.StartsWith(y)));

Try this:
myInitialList.RemoveAll(x =>myInitialList.Any(q => q != x && q.StartsWith(x)));
Or if you want to keep the original list, this is a way to get all the records that do not match your criteria:
List<string> resultList = myInitialList.Except(x => myInitialList.Any(q => q != x && q.StartsWith(x)));

How about
mylist = mylist.Where(a => mylist.All(b => b == a || !a.StartsWith(b)))
.Distinct()
.ToList();
This will return a new list where there isn't another item in the list that it starts with.
It has the extra check to allow returning the value where there string is the same, otherwise all items would be removed from the list.
Finally the distinct call means that two occurrences of the same string are removed.
Building on nsinreal's comment and solution you could do something like
myList = myList.OrderBy(d => d)
.Aggregate(new List<string>(),
(list, item) => {
if (!list.Any(x => item.StartsWith(x)))
list.Add(item);
return list;
}).ToList();
This reduces the complexity of the solution by reducing the size of the search list for each test. It still requires an initial sort.
Personally I find this alternative solution harder to read and my first answer is more expressive the problem to solve.

The most efficient way is IMO to sort the paths, then iterate them and return only the ones not starting as one of the previous, i.e. :
public static IEnumerable<string>
GetRootPathsOfSet(this IEnumerable<string> paths)
{
var sortedSet = new SortedSet<string>(paths,
StringComparer.CurrentCultureIgnoreCase);
string currRoot = null;
foreach (var p in sortedSet)
{
if (currRoot == null ||
!p.StartsWith(currRoot, StringComparison.InvariantCultureIgnoreCase))
{
currRoot = p;
yield return currRoot;
}
}
}
Some notes:
All the paths MUST terminate with a trailing back-slash, otherwise the StartsWith approach is not safe (e.g. C:\Dir and C:\Directory)
This code uses case-insensitive comparison
I'm not using pure LINQ here, but it's an extension method