We Keep Coding

Make using statement usable for multiple disposable objects

I have a bunch of text files in a folder, and all of them should have identical headers. In other words the first 100 lines of all files should be identical. So I wrote a function to check this condition:
private static bool CheckHeaders(string folderPath, int headersCount)
{
var enumerators = Directory.EnumerateFiles(folderPath)
.Select(f => File.ReadLines(f).GetEnumerator())
.ToArray();
//using (enumerators)
//{
for (int i = 0; i < headersCount; i++)
{
foreach (var e in enumerators)
{
if (!e.MoveNext()) return false;
}
var values = enumerators.Select(e => e.Current);
if (values.Distinct().Count() > 1) return false;
}
return true;
//}
}
The reason I am using enumerators is memory efficiency. Instead of loading all file contents in memory I enumerate the files concurrently line-by-line until a mismatch is found, or all headers have been examined.
My problem is evident by the commented lines of code. I would like to utilize a using block to safely dispose all the enumerators, but unfortunately using (enumerators) doesn't compile. Apparently using can handle only a single disposable object. I know that I can dispose the enumerators manually, by wrapping the whole thing in a try-finally block, and running the disposing logic in a loop inside finally, but is seems awkward. Is there any mechanism I could employ to make the using statement a viable option in this case?
Update
I just realized that my function has a serious flaw. The construction of the enumerators is not robust. A locked file can cause an exception, while some enumerators have already been created. These enumerators will not be disposed. This is something I want to fix. I am thinking about something like this:
var enumerators = Directory.EnumerateFiles(folderPath)
.ToDisposables(f => File.ReadLines(f).GetEnumerator());
The extension method ToDisposables should ensure that in case of an exception no disposables are left undisposed.

You can create a disposable-wrapper over your enumerators:
class DisposableEnumerable : IDisposable
{
private IEnumerable<IDisposable> items;
public event UnhandledExceptionEventHandler DisposalFailed;
public DisposableEnumerable(IEnumerable<IDisposable> items) => this.items = items;
public void Dispose()
{
foreach (var item in items)
{
try
{
item.Dispose();
}
catch (Exception e)
{
var tmp = DisposalFailed;
tmp?.Invoke(this, new UnhandledExceptionEventArgs(e, false));
}
}
}
}
and use it with the lowest impact to your code:
private static bool CheckHeaders(string folderPath, int headersCount)
{
var enumerators = Directory.EnumerateFiles(folderPath)
.Select(f => File.ReadLines(f).GetEnumerator())
.ToArray();
using (var disposable = new DisposableEnumerable(enumerators))
{
for (int i = 0; i < headersCount; i++)
{
foreach (var e in enumerators)
{
if (!e.MoveNext()) return false;
}
var values = enumerators.Select(e => e.Current);
if (values.Distinct().Count() > 1) return false;
}
return true;
}
}
The thing is you have to dispose those objects separately one by one anyway. But it's up to you where to encapsulate that logic. And the code I've suggested has no manual try-finally,)

To the second part of the question. If I get you right this should be sufficient:
static class DisposableHelper
{
public static IEnumerable<TResult> ToDisposable<TSource, TResult>(this IEnumerable<TSource> source,
Func<TSource, TResult> selector) where TResult : IDisposable
{
var exceptions = new List<Exception>();
var result = new List<TResult>();
foreach (var i in source)
{
try { result.Add(selector(i)); }
catch (Exception e) { exceptions.Add(e); }
}
if (exceptions.Count == 0)
return result;
foreach (var i in result)
{
try { i.Dispose(); }
catch (Exception e) { exceptions.Add(e); }
}
throw new AggregateException(exceptions);
}
}
Usage:
private static bool CheckHeaders(string folderPath, int headersCount)
{
var enumerators = Directory.EnumerateFiles(folderPath)
.ToDisposable(f => File.ReadLines(f).GetEnumerator())
.ToArray();
using (new DisposableEnumerable(enumerators))
{
for (int i = 0; i < headersCount; i++)
{
foreach (var e in enumerators)
{
if (!e.MoveNext()) return false;
}
var values = enumerators.Select(e => e.Current);
if (values.Distinct().Count() > 1) return false;
}
return true;
}
}
and
try
{
CheckHeaders(folderPath, headersCount);
}
catch(AggregateException e)
{
// Prompt to fix errors and try again
}

I'm going to suggest an approach that uses recursive calls to Zip to allow parallel enumeration of a normal IEnumerable<string> without the need to resort to using IEnumerator<string>.
bool Zipper(IEnumerable<IEnumerable<string>> sources, int take)
{
IEnumerable<string> ZipperImpl(IEnumerable<IEnumerable<string>> ss)
=> (!ss.Skip(1).Any())
? ss.First().Take(take)
: ss.First().Take(take).Zip(
ZipperImpl(ss.Skip(1)),
(x, y) => (x == null || y == null || x != y) ? null : x);
var matching_lines = ZipperImpl(sources).TakeWhile(x => x != null).ToArray();
return matching_lines.Length == take;
}
Now build up your enumerables:
IEnumerable<string>[] enumerables =
Directory
.EnumerateFiles(folderPath)
.Select(f => File.ReadLines(f))
.ToArray();
Now it's simple to call:
bool headers_match = Zipper(enumerables, 100);
Here's a trace of running this code against three files with more than 4 lines:
Ben Petering at 5:28 PM ACST
Ben Petering at 5:28 PM ACST
Ben Petering at 5:28 PM ACST
From a call 2019-05-23, James mentioned he’d like the ability to edit the current shipping price rules (eg in shipping_rules.xml) via the admin.
From a call 2019-05-23, James mentioned he’d like the ability to edit the current shipping price rules (eg in shipping_rules.xml) via the admin.
From a call 2019-05-23, James mentioned he’d like the ability to edit the current shipping price rules (eg in shipping_rules.xml) via the admin.
He also mentioned he’d like to be able to set different shipping price rules for a given time window, e.g. Jan 1 to Jan 30.
He also mentioned he’d like to be able to set different shipping price rules for a given time window, e.g. Jan 1 to Jan 30.
He also mentioned he’d like to be able to set different shipping price rules for a given time window, e.g. Jan 1 to Jan 30.
These storyishes should be considered when choosing the appropriate module to use.
These storyishes should be considered when choosing the appropriate module to use.X
These storyishes should be considered when choosing the appropriate module to use.
Note that the enumerations stop when they encountered a mismatch header in the 4th line on the second file. All enumerations then stopped.

Creating an IDisposable wrapper as #Alex suggested is correct. It needs just a logic to dispose already opened files if some of them is locked and probably some logic for error states. Maybe something like this (error state logic is very simple):
public class HeaderChecker : IDisposable
{
private readonly string _folderPath;
private readonly int _headersCount;
private string _lockedFile;
private readonly List<IEnumerator<string>> _files = new List<IEnumerator<string>>();
public HeaderChecker(string folderPath, int headersCount)
{
_folderPath = folderPath;
_headersCount = headersCount;
}
public string LockedFile => _lockedFile;
public bool CheckFiles()
{
_lockedFile = null;
if (!TryOpenFiles())
{
return false;
}
if (_files.Count == 0)
{
return true; // Not sure what to return here.
}
for (int i = 0; i < _headersCount; i++)
{
if (!_files[0].MoveNext()) return false;
string currentLine = _files[0].Current;
for (int fileIndex = 1; fileIndex < _files.Count; fileIndex++)
{
if (!_files[fileIndex].MoveNext()) return false;
if (_files[fileIndex].Current != currentLine) return false;
}
}
return true;
}
private bool TryOpenFiles()
{
bool result = true;
foreach (string file in Directory.EnumerateFiles(_folderPath))
{
try
{
_files.Add(File.ReadLines(file).GetEnumerator());
}
catch
{
_lockedFile = file;
result = false;
break;
}
}
if (!result)
{
DisposeCore(); // Close already opened files.
}
return result;
}
private void DisposeCore()
{
foreach (var item in _files)
{
try
{
item.Dispose();
}
catch
{
}
}
_files.Clear();
}
public void Dispose()
{
DisposeCore();
}
}
// Usage
using (var checker = new HeaderChecker(folderPath, headersCount))
{
if (!checker.CheckFiles())
{
if (checker.LockedFile is null)
{
// Error while opening files.
}
else
{
// Headers do not match.
}
}
}
I also removed .Select() and .Distinct() when checking the lines. The first just iterates over the enumerators array - the same as foreach above it, so you are enumerating this array twice. Then creates a new list of lines and .Distinct() enumerates over it.

Convert var into List<MapPdf>

I have a program that I used a variable of type List < MapPdf > that I will detach in variables when filling I would like to have the possibility to use it another time but I did not have the right to identify it again as
public static void Create(List<MapPdf> pps, Saison s, Agence agence)
{
foreach (var pelerins in grouped)
{
if (string.IsNullOrEmpty(pelerins.Key) || pelerins.Count() <= 0)
break;
if (writer.PageEvent == null)
{
writer.PageEvent = new Header_List()
{
travel = ctx.Travels.Include("Transport").First(v => v.UniqueId == pelerins.Key),
travelretour = ctx.Travels.Find(pelerins.First().UniqueIdRetour),
Agence = agence,
CountAllPelerin = pelerins.Count().ToString(),
CountFeminin = pelerins.Count(x => x.Sexe == PelerinSexe.Feminin).ToString(),
CountMasculin = pelerins.Count(x => x.Sexe == PelerinSexe.Masculin).ToString(),
NomGroupe = pelerins.First().Groupe,
NumeroDoc = writer.PageNumber
};
}
}
}
And i want to use pelerins as a List when i used in another function when it is of this declaration
I used List < MapPdf > pls = pelerins.ToList(); but it does not work
CreateFr(pls, false, cb, s);

If you are referring to var pelerins within the for-each loop and if I understand the problem, you are unable use it into another method , because var pelerins is a local variable encapsulated within the for-each loop -It does not exist outside it.
You could do the following:
//public property to retrieve pelerins
public List <MapPdf> pls new List<MapPdf>();
...
...
public static void Create(List<MapPdf> pps, Saison s, Agence agence)
{
foreach (var pelerins in grouped)
{
if (string.IsNullOrEmpty(pelerins.Key) || pelerins.Count() <= 0)
break;
if (writer.PageEvent == null)
{
//do logic
...
//store the one you are interested in, so you can use it later on
pls = pelerins.ToList();
}
}
}

C# Filter Items In A List According To Multiple Criteria

First, what my situation here is...
My SomeObject has a property string Status which I am interested in for this scenario.
Status property can contain "Open", "Closed", "Finished" values exactly.
I have a method called FilterObjects which returns a List<SomeObject>
Method accepts an argument same as its return type, List<SomeObject>
Method is supposed to filter according to following cases explained below and return the list of objects.
The List<SomeObject> I am sending as argument to my method is guaranteed to be in order (through their ID and type).
The cases are (all related to the string Status property I mentioned):
If any item in the list contains Status = "Finished"; then eliminate all other elements that was in the original list and return only the object that has the "Finished" status.
If any item does NOT contain Status = Finished but contains "CLOSED", I need to check if there is any other item that has the value of "Open" after that "CLOSED" one. You can think of this as a "a task can be closed, but can be reopened. But once it is finished, it cannot be reopened".
If it contains a "CLOSED" and does not have any "OPEN" after that item, I will ignore all the items before CLOSED and only return CLOSED object. If it contains "OPEN" after any closed, I need to return anything AFTER that CLOSED, by excluding itself.
I also tried explain the same thing with my awesome MS Paint skills.
The object itself is not really a problem, but my method is something like this:
private List<SomeObject> FilterObjects(List<SomeObject> objectList)
{
var objects = objectList;
var returnList = new List<SomeObject>();
foreach (var obj in objects)
{
if (obj.Status == "Finished")
{
returnList.Add(obj);
return returnList;
}
}
return new List<SomeObject>();
}
Long story short, what would be the best and most efficient way to apply all this logic in this single method? Honestly, I couldn't go further than the first case I already implemented, which is the FINISHED. Could this whole thing be done with some LINQ magic?
It is guaranteed that I receive an ordered list AND I will never get items more than a couple of hundred so the collection will never be massive.
Many thanks in advance for the help.

You can try something like that:
private List<SomeObject> FilterObjects(List<SomeObject> objectList)
{
SomeObject finished = objectList.FirstOrDefault(o => o.Status.Equals("Finished"));
if (finished != null) { return new List<SomeObject> { finished }; }
List<SomeObject> closed = objectList.SkipWhile(o => !o.Status.Equals("Closed")).ToList();
if (closed.Count == 1) { return closed; }
if (closed.Count > 1) { return closed.Skip(1).ToList(); }
// if you need a new list object than return new List<SomeObject>(objectList);
return objectList;
}

I really wouldn't bother using Linq for this, as you will either create an overly complicated instruction to manage or you will require several loop iterations. I would go for something like this instead:
private List<SomeObject> FilterObjects(List<SomeObject> objectList)
{
int lastClosed = -1;
for (int i = 0; i < objectList.Count; i++)
{
if (objectList[i].Status == "Closed")
lastClosed = i;
else if (objectList[i].Status == "Finished")
return new List<SomeObject>() { objectList[i] };
}
if (lastClosed > -1)
if (lastClosed == objectList.Count - 1)
return new List<SomeObject>() { objectList[lastClosed] };
else
return objectList.Skip(lastClosed + 1).ToList();
else
return objectList;
}
EDIT: slightly changed the last bit of code so that it won't trigger an exception if the objectList is empty

LINQ is not well suited and inefficient for scenarios where you need to apply logic based on previous / next elements of a sequence.
The optimal way to apply your logic is to use a single loop and track the Closed status and the position where the status change occurred. At the end you'll return a single element at that position if the last status is Closed, or a range starting at that position otherwise.
static List<SomeObject> FilterObjects(List<SomeObject> objectList)
{
int pos = 0;
bool closed = false;
for (int i = 0; i < objectList.Count; i++)
{
var item = objectList[i];
if (item.Status == "Finished")
return new List<SomeObject> { item };
if (item.Status == (closed ? "Opened" : "Closed"))
{
pos = i;
closed = !closed;
}
}
return objectList.GetRange(pos, closed ? 1 : objectList.Count - pos);
}

I did it this way:
public static IEnumerable<SomeObject> convert(this IEnumerable<SomeObject> input)
{
var finished = input.FirstOrDefault(x => x.Status == "Finished");
if (finished != null)
{
return new List<SomeObject> {finished};
}
return input.Aggregate(new List<SomeObject>(), (a, b) =>
{
if (!a.Any())
{
a.Add(b);
}
else if (b.Status == "Open")
{
if (a.Last().Status == "Closed")
{
a.Remove(a.Last());
}
a.Add(b);
}
else if (b.Status == "Closed")
{
a = new List<SomeObject> {b};
}
return a;
});
}

You can write a method like this. This is bare minimum you will have to add null check and exception handling.
public List<SomeCls> GetResult(List<SomeCls> lstData)
{
List<SomeCls> lstResult;
if(lstData.Any(x=>x.Status=="Finished"))
{
lstResult = lstData.Where(x=>x.Status=="Finished").ToList();
}
else if(lstData.Any(x=>x.Status=="Closed"))
{
// Here assuming that there is only one Closed in whole list
int index = lstData.FindIndex(0,lstData.Count(),x=>x.Status=="Closed");
lstResult = lstData.GetRange(index,lstData.Count()-index);
if(lstResult.Count()!=1) // check if it contains Open.
{
lstResult = lstResult.Where(x=>x.Status=="Open").ToList();
}
}
else // Only Open
{
lstResult = lstData;
}
return lstResult;
}

something like this :
private List<SomeObject> FilterObjects(List<SomeObject> objectList)
{
if (objectList.Where(x => x.Status == "Finished").Any())
{
return objectList.Where(x => x.Status == "Finished").ToList();
}
else if (objectList.Where(x => x.Status == "Closed").Any())
{
if (objectList.FindIndex(x => x.Status == "Closed") == objectList.Count() - 1)
{
return objectList.Where(x => x.Status == "Closed").ToList();
}
else
{
return objectList.GetRange(objectList.FindIndex(x => x.Status == "Closed") + 1, objectList.Count() - (objectList.FindIndex(x => x.Status == "Closed") + 1));
}
}
return objectList;
}

Taking a set of items from a list and to perform some operation based on need

I have a list of items in a list. From that list I need to take the first 1000 items and need to submit the package and then again I need to take another 1000 and need to submit a package. If the list is not having 1000 I need to submit the package with all the items. for that I wrote the following code which is returning an error as collection modified.
List<SyncQueue> tempMassiveSyncQueue=massiveSyncQueue;
while (tempMassiveSyncQueue.Count != 0)
{
int MassivePackageFileCount =Convert.ToInt32(ConfigurationManager.AppSettings["MassivePackageFileLimit"]);
massiveSyncQueues = tempMassiveSyncQueue;
List<SyncQueue> tempMassivePackageSyncQueue=new List<SyncQueue>();
if (massiveSyncQueues.Count > 1000
{
var massivePackageSyncQueue = (massiveSyncQueues.Take(1000)).ToList<SyncQueue>();
tempMassivePackageSyncQueue = massivePackageSyncQueue;
SubmitPackage(massivePackageSyncQueue);
}
if (tempMassivePackageSyncQueue.Count != 0)
{
foreach (var massivesynq in tempMassiveSyncQueue)
{
foreach (var deleteId in tempMassivePackageSyncQueue.Where(id => id.SyncQueueId == massivesynq.SyncQueueId))
{
tempMassiveSyncQueue.Remove(massivesynq);
}
}
}
else
{
SubmitPackage(massiveSyncQueues);
}
massiveSyncQueues = null;
}
Can any one help on this?

Incorporate Skip into your logic
int loopCount = 0;
While(true)
{
var ListToProcess = massiveSyncQueue.Skip(loopCount*1000).Take(1000);
SubmitPackage(ListToProcess);
if(ListToProcess.Count < 1000) // We know there are no more in the list massive list.
{
break;
}
loopCnt++;
}

Your problem is that you are adjusting the collection on which the bounds of the foreach construct are set.
Try using ToList() on the collection you are looping through, as this creates a new List in memory:
foreach (var massivesynq in tempMassiveSyncQueue.ToList())
{
foreach (var deleteId in tempMassivePackageSyncQueue.Where(id => id.SyncQueueId == massivesynq.SyncQueueId).ToList())
{
tempMassiveSyncQueue.Remove(massivesynq);
}
}

In line 1 you set tempMassiveSyncQueue = massiveSyncQueue, yet within the while loop you set massiveSyncQueue = tempMassiveSync.
The collection modified error usually occurs when you modify the collection you are looping through. Which is why you need to first create a copy of the collection which is INDEPENDANT to the original collection and loop through that.
Before the while loop you need to add all items in massiveSyncQueue to tempMassiveSyncQueue. You then need to loop through the temp list with your code. In the second loop, you are removing items from the list you are looping through. I assume you meant to remove the items from massiveSyncQueue and not the temp list.
Try the following:
List<SyncQueue> tempMassiveSyncQueue = new List<SyncQueue>();
foreach(var item in massiveSyncQueue)
{
tempMassiveSyncQueue.Add(item);
}
while (tempMassiveSyncQueue.Count != 0)
{
int MassivePackageFileCount = Convert.ToInt32(ConfigurationManager.AppSettings["MassivePackageFileLimit"]);
List<SyncQueue> tempMassivePackageSyncQueue=new List<SyncQueue>();
if (massiveSyncQueues.Count > 1000
{
var massivePackageSyncQueue = (massiveSyncQueues.Take(1000)).ToList<SyncQueue>();
tempMassivePackageSyncQueue = massivePackageSyncQueue;
SubmitPackage(massivePackageSyncQueue);
}
if (tempMassivePackageSyncQueue.Count != 0)
{
foreach (var massivesynq in massiveSyncQueue)
{
foreach (var deleteId in tempMassivePackageSyncQueue.Where(id => id.SyncQueueId == massivesynq.SyncQueueId))
{
massiveSyncQueue.Remove(massivesynq);
}
}
}
else
{
SubmitPackage(massiveSyncQueues);
}
massiveSyncQueues = null;
}

Try this
int count=1;
while(tempMassivePackageSyncQueue.Count>1000)
{
var massivePackageSyncQueue = (massiveSyncQueues.skip(count*1000).Take(1000)).ToList<SyncQueue>();
tempMassivePackageSyncQueue = massivePackageSyncQueue;
SubmitPackage(massivePackageSyncQueue);
count++;
}
var massivePackageSyncQueue = (massiveSyncQueues.skip(count*1000).Take()).ToList<SyncQueue>();
tempMassivePackageSyncQueue = massivePackageSyncQueue;
SubmitPackage(massivePackageSyncQueue);

List<T>Get Chunk Number being executed

I am breaking a list into chunks and processing it as below:
foreach (var partialist in breaklistinchunks(chunksize))
{
try
{
do something
}
catch
{
print error
}
}
public static class IEnumerableExtensions
{
public static IEnumerable<List<T>> BreakListinChunks<T>(this IEnumerable<T> sourceList, int chunkSize)
{
List<T> chunkReturn = new List<T>(chunkSize);
foreach (var item in sourceList)
{
chunkReturn.Add(item);
if (chunkReturn.Count == chunkSize)
{
yield return chunkReturn;
chunkReturn = new List<T>(chunkSize);
}
}
if (chunkReturn.Any())
{
yield return chunkReturn;
}
}
}
If there is an error, I wish to run the chunk again. Is it possible to find the particular chunk number where we received the error and run that again ?
The batches have to be executed in sequential order .So if batch#2 generates an error, then I need to be able to run 2 again, if it fails again. I just need to get out of the loop for good .

List<Chunk> failedChunks = new List<Chunk>();
foreach (var partialist in breaklistinchunks(chunksize))
{
try
{
//do something
}
catch
{
//print error
failedChunks.Add(partiallist);
}
}
// attempt to re-process failed chunks here

I propose this answer based on your comment to Aaron's answer.
The batches have to be executed in sequential order .So if 2 is a problem , then I need to be able to run 2 again, if it fails again. I just need to get out of the loop for good.
foreach (var partialist in breaklistinchunks(chunksize))
{
int fails = 0;
bool success = false;
do
{
try
{
// do your action
success = true; // should be on the last line before the 'catch'
}
catch
{
fails += 1;
// do something about error before running again
}
}while (!success && fails < 2);
// exit the iteration if not successful and fails is 2
if (!success && fails >= 2)
break;
}

I made a possible solution for you if you don't mind switching from Enumerable to Queue, which kind of fits given the requirements...
void Main()
{
var list = new Queue<int>();
list.Enqueue(1);
list.Enqueue(2);
list.Enqueue(3);
list.Enqueue(4);
list.Enqueue(5);
var random = new Random();
int chunksize = 2;
foreach (var chunk in list.BreakListinChunks(chunksize))
{
foreach (var item in chunk)
{
try
{
if(random.Next(0, 3) == 0) // 1 in 3 chance of error
throw new Exception(item + " is a problem");
else
Console.WriteLine (item + " is OK");
}
catch (Exception ex)
{
Console.WriteLine (ex.Message);
list.Enqueue(item);
}
}
}
}
public static class IEnumerableExtensions
{
public static IEnumerable<List<T>> BreakListinChunks<T>(this Queue<T> sourceList, int chunkSize)
{
List<T> chunkReturn = new List<T>(chunkSize);
while(sourceList.Count > 0)
{
chunkReturn.Add(sourceList.Dequeue());
if (chunkReturn.Count == chunkSize || sourceList.Count == 0)
{
yield return chunkReturn;
chunkReturn = new List<T>(chunkSize);
}
}
}
}
Outputs
1 is a problem
2 is OK
3 is a problem
4 is a problem
5 is a problem
1 is a problem
3 is OK
4 is OK
5 is OK
1 is a problem
1 is OK

One possibility would be to use a for loop instead of a foreach loop and use the counter as a means to determine where an error occurred. Then you could continue from where you left off.

You can use break to exit out of the loop as soon as a chunk fails twice:
foreach (var partialList in breaklistinchunks(chunksize))
{
if(!TryOperation(partialList) && !TryOperation(partialList))
{
break;
}
}
private bool TryOperation<T>(List<T> list)
{
try
{
// do something
}
catch
{
// print error
return false;
}
return true;
}
You could even make the loop into a one-liner with LINQ, but it is generally bad practice to combine LINQ with side-effects, and it's not very readable:
breaklistinchunks(chunksize).TakeWhile(x => TryOperation(x) || TryOperation(x));