We Keep Coding

c# multi threading process large file lines in batches of 100 [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have a file with 500.000.000 lines.
The lines are string of max 10 characters.
How can I process this file using multi threading and in batches of 100?

Using MoreLinq's Batch method, this will create a collection of IEnumerable<string> which will contain the line batch size of 100, it will spin a new task for every 100 lines.
This is a basic implementation, it might be wise to use a Semaphore to only run a certain amount of tasks at any given time, and also seeing what overhead File.ReadAllLines will have on performance with 500,000,000 lines.
public class FileProcessor
{
public async Task ProcessFile()
{
List<Task> tasks = new List<Task>();
var lines = File.ReadAllLines("File.txt").Batch(100);
foreach (IEnumerable<string> linesBatch in lines)
{
IEnumerable<string> localLinesBatch = linesBatch;
Task task = Task.Factory.StartNew(() =>
{
// Perform operation on localLinesBatch
});
tasks.Add(task);
}
await Task.WhenAll(tasks);
}
}
public static class LinqExtensions
{
public static IEnumerable<IEnumerable<TSource>> Batch<TSource>(
this IEnumerable<TSource> source, int size)
{
TSource[] bucket = null;
var count = 0;
foreach (var item in source)
{
if (bucket == null)
bucket = new TSource[size];
bucket[count++] = item;
if (count != size)
continue;
yield return bucket;
bucket = null;
count = 0;
}
if (bucket != null && count > 0)
yield return bucket.Take(count);
}
}

Using additional libraries is not required if you use Parallel.ForEach from built-in TPL and write a couple of enumerators (listed below). Your code can look like this:
using (var input = new StreamReader(File.OpenRead(#"c:\path\to\my\file.txt")))
{
Parallel.ForEach(
input.ReadLines().TakeChunks(100),
new ParallelOptions() { MaxDegreeOfParallelism = 8 /* better be number of CPU cores */ },
batchOfLines => {
DoMyProcessing(batchOfLines);
});
}
for this to work, you need a couple of extension methods on IEnumerable<T> and a couple of enumerators, defined as follows:
public static class EnumerableExtensions
{
public static IEnumerable<string> ReadLines(this StreamReader input)
{
return new LineReadingEnumerable(input);
}
public static IEnumerable<IReadOnlyList<T>> TakeChunks<T>(this IEnumerable<T> source, int length)
{
return new ChunkingEnumerable<T>(source, length);
}
public class LineReadingEnumerable : IEnumerable<string>
{
private readonly StreamReader _input;
public LineReadingEnumerable(StreamReader input)
{
_input = input;
}
public IEnumerator<string> GetEnumerator()
{
return new LineReadingEnumerator(_input);
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public class LineReadingEnumerator : IEnumerator<string>
{
private readonly StreamReader _input;
private string _current;
public LineReadingEnumerator(StreamReader input)
{
_input = input;
}
public void Dispose()
{
_input.Dispose();
}
public bool MoveNext()
{
_current = _input.ReadLine();
return (_current != null);
}
public void Reset()
{
throw new NotSupportedException();
}
public string Current
{
get { return _current; }
}
object IEnumerator.Current
{
get { return _current; }
}
}
public class ChunkingEnumerable<T> : IEnumerable<IReadOnlyList<T>>
{
private readonly IEnumerable<T> _inner;
private readonly int _length;
public ChunkingEnumerable(IEnumerable<T> inner, int length)
{
_inner = inner;
_length = length;
}
public IEnumerator<IReadOnlyList<T>> GetEnumerator()
{
return new ChunkingEnumerator<T>(_inner.GetEnumerator(), _length);
}
IEnumerator IEnumerable.GetEnumerator()
{
return this.GetEnumerator();
}
}
public class ChunkingEnumerator<T> : IEnumerator<IReadOnlyList<T>>
{
private readonly IEnumerator<T> _inner;
private readonly int _length;
private IReadOnlyList<T> _current;
private bool _endOfInner;
public ChunkingEnumerator(IEnumerator<T> inner, int length)
{
_inner = inner;
_length = length;
}
public void Dispose()
{
_inner.Dispose();
_current = null;
}
public bool MoveNext()
{
var currentBuffer = new List<T>();
while (currentBuffer.Count < _length && !_endOfInner)
{
if (!_inner.MoveNext())
{
_endOfInner = true;
break;
}
currentBuffer.Add(_inner.Current);
}
if (currentBuffer.Count > 0)
{
_current = currentBuffer;
return true;
}
_current = null;
return false;
}
public void Reset()
{
_inner.Reset();
_current = null;
_endOfInner = false;
}
public IReadOnlyList<T> Current
{
get
{
if (_current != null)
{
return _current;
}
throw new InvalidOperationException();
}
}
object IEnumerator.Current
{
get
{
return this.Current;
}
}
}
}

How to make my InfiniteLoopingList class implement IEnumerable?

I am making a prototype application and for that I designed a class that behaves like an infinite looping list. That is, if my internal list contains 100 values, when I ask for the 101st value, I get the first, the 102nd yields the second, and so on, repeating.
So I would like to write the following code:
var slice = loopingListInstance.Skip(123).Take(5);
And for that I need to implement IEnumerable suitable, as I understand.
Here is my current code:
public class InfiniteLoopingList : IEnumerable<double>
{
double[] _values = File.ReadLines(#"c:\file.txt")
.Select(s => double.Parse(s, CultureInfo.InvariantCulture))
.ToArray();
int _size;
public InfiniteLoopingList()
{
_size = _values.Length;
}
public double this[int i]
{
get { return _values[i % _size]; }
set { _values[i % _size] = value; }
}
public IEnumerator<double> GetEnumerator()
{
return this.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
// ???? now what ?? :(
}
}

Since you implemented the indexer property, you could do it via the simplest way as follows:
public IEnumerator<double> GetEnumerator()
{
int i = 0;
while (true)
yield return this[i++];
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
EDIT
Please notice, that this is not really infinite loop. This approach will only work until i = int.MaxValue. Thanks to #oleksii.

You don't need a class for this...
An extension method will do the trick:
public static class InfEx
{
public static IEnumerable<T> LoopForever<T>(this IEnumerable<T> src)
{
var data = new List<T>();
foreach(var item in src)
{
data.Add(item);
yield return item;
}
for(;;)
{
foreach(var item in data)
{
yield return item;
}
}
}
}
Now you can take a sequence and make it a looping, infinite sequence:
IEnumerable<Foo> mySeq = ...;
IEnumerable<Foo> infMySeq = mySeq.LoopForver();
IEnumerable<Foo> aSelectionOfInfMySeq = infMySeq.Skip(101).Take(5);

You can implement the IEnumerator interface:
class InifniteEnumerator<T> : IEnumerator<T> {
private int index = -1;
private IList<T> innerList;
private int repeatPos;
public InifniteEnumerator(IList<T> innerList, int repeatPos) {
this.innerList = innerList;
this.repeatPos = repeatPos;
}
public T Current {
get {
if (index == -1) {
throw new InvalidOperationException();
}
return this.innerList[index];
}
}
object IEnumerator.Current {
get {
return this.Current;
}
}
public void Dispose() {
}
public bool MoveNext() {
this.index++;
if (this.index == repeatPos) {
this.index = 0;
}
return true;
}
public void Reset() {
this.index = -1;
}
}
and then return an instance of it in the GetEnumerator methods:
IEnumerator IEnumerable.GetEnumerator() {
return this.GetEnumerator();
}
public IEnumerator<T> IEnumerable<T>.GetEnumerator() {
return new InifniteEnumerator(this, 100);
}

call graph IEnumerator

I have to implement a call graph for expressions like Id = Id(Param); and that wasn't a problem.
Now I have to implement an enumerator which lists one at a time all topological orderings among the calls that satisfy the order of dependencies.
And here's the trouble.
This is a simple node for the call graph:
class CallGraphNode
{
private string name;
public List<CallGraphNode> dependents = new List<CallGraphNode>();
public int dependencies;
private bool executed = false;
public bool Executable { get { return dependencies == 0; } }
public bool Executed { get { return executed; } set { executed = value; } }
public CallGraphNode(string name)
{
this.name = name;
dependencies = 0;
}
public override string ToString()
{
return name;
}
public void AddDependent(CallGraphNode n)
{
dependents.Add(n);
}
}
And this is the call graph class itself:
class CallGraph : IEnumerable<List<CallGraphNode>>
{
public List<CallGraphNode> nodes = new List<CallGraphNode>();
public void AddNode(CallGraphNode n)
{
nodes.Add(n);
}
public static void Show(IEnumerable<CallGraphNode> n)
{
foreach (CallGraphNode node in n)
{
Console.Write("{0} ", node);
}
Console.WriteLine();
}
static IEnumerable<List<CallGraphNode>> EnumerateFunctions(List<CallGraphNode> executable, List<CallGraphNode> res)
{
if (executable.Count == 0)
yield return res;
else foreach (CallGraphNode n in executable)
{
if (!n.Executed)
res.Add(n);
List<CallGraphNode> next_executable = new List<CallGraphNode>(executable);
executable.Remove(n);
foreach (CallGraphNode m in n.dependents)
if (--m.dependencies == 0)
next_executable.Add(m);
foreach (List<CallGraphNode> others in EnumerateFunctions(next_executable, res))
yield return others;
foreach (CallGraphNode m in n.dependents)
m.dependencies++;
if (!n.Executed)
res.Remove(n);
}
}
IEnumerator<List<CallGraphNode>> IEnumerable<List<CallGraphNode>>.GetEnumerator()
{
List<CallGraphNode> executable = new List<CallGraphNode>();
foreach (CallGraphNode n in nodes)
if (n.Executable || n.Executed)
executable.Add(n);
List<CallGraphNode> output = new List<CallGraphNode>();
foreach (List<CallGraphNode> list in EnumerateFunctions(executable, output))
yield return list;
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{ throw new NotImplementedException(); }
}
Now, the problem is it just won't work. When I try to create an IEnumerator and assign it the GetEnumerator() return value, I get a casting error and that's honestly what I expected when trying to do so:
IEnumerator<List<CallGraphNode>> lt = cg.GetEnumerator();
Then I've tried:
System.Collections.Generic.List<CallGraphNode>.Enumerator en = cg.nodes.GetEnumerator();
This works, but the method EnumerateFunctions is never called and the enumerator just contains the original list of graph nodes.
Any ideas?

The problem is that you're implementing both IEnumerable<T> and IEnumerable using explicit interface implementation.
You probably want to change this declaration:
IEnumerator<List<CallGraphNode>> IEnumerable<List<CallGraphNode>>.GetEnumerator()
to be a "normal" interface implementation:
public IEnumerator<List<CallGraphNode>> GetEnumerator()
Alternatively, you could stick with explicit interface implementation, but use:
IEnumerable<List<CallGraphNode>> sequence = cg;
IEnumerator<List<CallGraphNode>> lt = sequence.GetEnumerator();

C#: Circular enumeration of IEnumerable<T>

There is the command hierarchy in my current application.
public interface ICommand
{
void Execute();
}
So, some commands are stateful, some are not.
I need to enumerate IEnumerable in the circular way for some command implementation during command execution.
public class GetNumberCommand : ICommand
{
public GetNumberCommand()
{
List<int> numbers = new List<int>
{
1, 2, 3
};
}
public void Execute()
{
// Circular iteration here.
// 1 => 2 => 3 => 1 => 2 => 3 => ...
}
public void Stop()
{
// Log current value. (2 for example)
}
}
Execute is called from time to time, so it is necessary to store the iteration state.
How to implement that circular enumeration?
I have found two solutions:
Using the IEnumerator<T> interface.
It looks like:
if (!_enumerator.MoveNext())
{
_enumerator.Reset();
_enumerator.MoveNext();
}
Using the circular IEnumerable<T> (yield forever the same sequence): “Implementing A Circular Iterator” - HonestIllusion.Com.
Maybe, there are more ways to achieve it.
What would you recommend to use and why?

Instead of dealing with IEnumerator interface,
foreach (var x in GetSomething())
{
if (someCondition) break;
}
public IEnumerable<int> GetSomething()
{
List<int> list = new List<int>() { 1, 2, 3 };
int index=0;
while (true)
yield return list[index++ % list.Count];
}

Here's one I just implemented as an extension.
using System.Collections;
using System.Collections.Generic;
using System.Linq;
namespace DroopyExtensions
{
public static class CircularEnumaratorExtensionMethod
{
public static IEnumerator<T> GetCircularEnumerator<T>(this IEnumerable<T> t)
{
return new CircularEnumarator<T>(t.GetEnumerator());
}
private class CircularEnumarator<T> : IEnumerator<T>
{
private readonly IEnumerator _wrapedEnumerator;
public CircularEnumarator(IEnumerator wrapedEnumerator)
{
this._wrapedEnumerator = wrapedEnumerator;
}
public object Current => _wrapedEnumerator.Current;
T IEnumerator<T>.Current => (T)Current;
public void Dispose()
{
}
public bool MoveNext()
{
if (!_wrapedEnumerator.MoveNext())
{
_wrapedEnumerator.Reset();
return _wrapedEnumerator.MoveNext();
}
return true;
}
public void Reset()
{
_wrapedEnumerator.Reset();
}
}
}
}
To use it, all you have to do is
using DroopyExtensions;
class Program
{
static void Main(string[] args)
{
var data = new List<string>() {"One", "Two", "Tree"};
var dataEnumerator = data.GetCircularEnumerator();
while(dataEnumerator.MoveNext())
{
Console.WriteLine(dataEnumerator.Current);
}
}
}

You can use this extension method:
public static IEnumerable<T> Cyclic<T>(this IEnumerable<T> #this)
{
while (true)
foreach (var x in #this)
yield return x;
}
In that way:
public class GetNumberCommand : ICommand
{
private readonly IEnumerator<int> _commandState = new[] { 1, 2, 3 }.Cyclic().GetEnumerator();
public void Execute()
{
_commandState.MoveNext();
var state = _commandState.Current;
//
// Do stuff with state
//
}
public void Stop()
{
var state = _commandState.Current;
// Log state value. (2 for example)
}
}

while (!stop)
{
foreach (var i in numbers)
{
// do something
}
}

I think, the most comfortable way wil be to implement custom collection with custom enumerator and encapsulate circular logic in it.
class Collection<T> : IEnumerable<T>
{
bool circle;
List<T> collection = new List<T>();
public IEnumerable<T> IEnumerable<T>.GetEnumerator()
{
if(circle) return new CustomEnumerator<T>(this);
return circle.GetEnumerator();
}
}
class CustomEnumerator : Enumerator<T> {}
something like this...

You can write a circular enumerable without yield returns.
public class CircularEnumerable<T> : IEnumerable<T>
{
public CircularEnumerable (IEnumerable<T> sequence)
{
InfiniteLoop = sequence.Concat (this);
}
private readonly IEnumerable<T> InfiniteLoop;
public IEnumerator<T> GetEnumerator ()
{
return InfiniteLoop.GetEnumerator ();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator ()
{
return InfiniteLoop.GetEnumerator ();
}
}
public class GetNumberCommand : ICommand
{
public GetNumberCommand()
{
List<int> numbers = new List<int>
{
1, 2, 3
};
infiniteLoopOnNumbers = new CircularEnumerable<int>(numbers).GetEnumerator();
}
IEnumerator<int> infiniteLoopOnNumbers;
public void Execute()
{
infiniteLoopOnNumbers.MoveNext();
}
public void Stop()
{
int value = infiniteLoopOnNumbers.Current;
}
}

Is it possible to have C++-like generics in C#?

Perhaps this question was already asked million times but I couldn't find similar topic. Is it possible to write a generic class with multiple 'where'-s that will be checked in compile time? Here is what I have today
public class MsBitsEnumWrapper<TC, TE> : IEnumerable<TC>
{
internal class Helper : IEnumerator<TC>
{
private readonly TC[] _data;
private int _pos = -1;
private readonly IEnumBackgroundCopyJobs _jobs;
private readonly IEnumBackgroundCopyFiles _files;
// I miss C++ templates that allows me to implements this method in completelly generic fashion...
public Helper(TE source)
{
_jobs = source as IEnumBackgroundCopyJobs;
_files = source as IEnumBackgroundCopyFiles;
uint count = 0;
if (null != _jobs)
{
_jobs.GetCount(out count);
_jobs.Reset();
}
else
if (null != _files)
{
_files.GetCount(out count);
_files.Reset();
}
_data = new TC[count];
for (uint i = 0; i < count; ++i)
{
uint fetched = 0;
if (null != _jobs)
{
IBackgroundCopyJob job;
_jobs.Next(1, out job, ref fetched);
_data[i] = (TC)job;
}
else
if (null != _files)
{
IBackgroundCopyFile file;
_files.Next(1, out file, ref fetched);
_data[i] = (TC)file;
}
}
}
#region Implementation of IDisposable
public void Dispose() { }
#endregion
#region Implementation of IEnumerator
public bool MoveNext()
{
if (_pos < (_data.Length - 1))
{
_pos++;
return true;
}
return false;
}
public void Reset()
{
_pos = -1;
}
public TC Current
{
get { return _data[_pos]; }
}
object IEnumerator.Current
{
get { return Current; }
}
#endregion
}
private readonly Helper _enumerator;
public MsBitsEnumWrapper(TE source) { _enumerator = new Helper(source); }
#region Implementation of IEnumerable
public IEnumerator<TC> GetEnumerator() { return _enumerator; }
IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); }
#endregion
}
Obviously I don't like it because I have to have switch in run-time that detect the type of generic argument. Is it possible to have something like this ?
public class MsBitsEnumWrapper<TC, TE> : IEnumerable<TC>
where TE : IEnumBackgroundCopyJobs, IEnumBackgroundCopyFiles
where TC : IBackgroundCopyJob, IBackgroundCopyFile
{
internal class Helper : IEnumerator<TC>
{
private readonly TC[] _data;
private int _pos = -1;
private readonly TE _iface;
// I miss C++ templates that allows me to implements this method in completelly generic fashion...
public Helper(TE source)
{
_iface = source;
uint count;
_iface.GetCount(out count);
_iface.Reset();
_data = new TC[count];
for (uint i = 0; i < count; ++i)
{
uint fetched = 0;
TC job;
_iface.Next(1, out job, ref fetched);
_data[i] = job;
}
}
#region Implementation of IDisposable
public void Dispose() { }
#endregion
#region Implementation of IEnumerator
public bool MoveNext()
{
if (_pos < (_data.Length - 1))
{
_pos++;
return true;
}
return false;
}
public void Reset()
{
_pos = -1;
}
public TC Current
{
get { return _data[_pos]; }
}
object IEnumerator.Current
{
get { return Current; }
}
#endregion
}
private readonly Helper _enumerator;
public MsBitsEnumWrapper(TE source) { _enumerator = new Helper(source); }
#region Implementation of IEnumerable
public IEnumerator<TC> GetEnumerator() { return _enumerator; }
IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); }
#endregion
}
I realize that it won't work if generic is defined with TE=IEnumBackgroundCopyJobs and TC=IBackgroundCopyFile, but since I'm the one who defines TE and TC and function prototypes of given interfaces are the same it would be nice to have it work this way.
Or may be there is another way to simplify current code ?
Thanks!

As JaredPar indicated, you can't do this in C#. The choices are to use delegates, or go old school with using an abstract base class.
My implementation below take advantage of 'yield return' to cut down on the implementation of IEnumerable.
#if USE_DELEGATES
public class MsBitsEnum<T> : IEnumerable<T>
{
Action _reset;
Func<T> _next;
Func<int> _count;
public MsBitsEnum(Action reset, Func<T> next, Func<int> count)
{
_reset = reset;
_next = next;
_count = count;
}
public IEnumerator<T> GetEnumerator()
{
_reset();
int count = _count();
for (int i = 0; i < count; ++i)
yield return _next();
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public class MsBitsJobs : MsBitsEnum<IBackgroundCopyJob>
{
public MsBitsJobs(IEnumBackgroundCopyJobs jobs)
: base(() => jobs.Reset(),
() =>
{
IBackgroundCopyJob job = null;
uint fetched = 0;
jobs.Next(1, out job, out fetched);
return fetched == 1 ? job : null;
},
() =>
{
uint count;
jobs.GetCount(out count);
return (int) count;
})
{
}
}
public class MsBitsFiles : MsBitsEnum<IBackgroundCopyFile>
{
public MsBitsFiles(IEnumBackgroundCopyFiles files)
: base(() => files.Reset(),
() =>
{
IBackgroundCopyFile file = null;
uint fetched = 0;
files.Next(1, out file, out fetched);
return fetched == 1 ? file : null;
},
() =>
{
uint count;
files.GetCount(out count);
return (int)count;
})
{
}
}
#else // !USE_DELEGATES
public abstract class MsBitsEnum<T> : IEnumerable<T>
{
protected abstract void Reset();
protected abstract bool Next(out T item);
public IEnumerator<T> GetEnumerator()
{
T item;
Reset();
while (Next(out item))
yield return item;
}
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public class MsBitsJobs : MsBitsEnum<IBackgroundCopyJob>
{
IEnumBackgroundCopyJobs _jobs;
protected override void Reset()
{
_jobs.Reset();
}
protected override bool Next(out IBackgroundCopyJob job)
{
uint fetched;
_jobs.Next(1, out job, out fetched);
return fetched == 1;
}
public MsBitsJobs(IEnumBackgroundCopyJobs jobs)
{
_jobs = jobs;
}
}
public class MsBitsFiles : MsBitsEnum<IBackgroundCopyFile>
{
IEnumBackgroundCopyFiles _files;
protected override void Reset()
{
_files.Reset();
}
protected override bool Next(out IBackgroundCopyFile file)
{
uint fetched;
_files.Next(1, out file, out fetched);
return fetched == 1;
}
public MsBitsFiles(IEnumBackgroundCopyFiles files)
{
_files = files;
}
}
#endif // !USE_DELEGATES

It sounds like what you're asking is if C# supports structural typing in generics in the same way that C++ does for templates. The answer is No.
C++ templates aren't fully evaluated for correctness until the member functions are used with a given generic instantiation. C# generics on the other hand are fully evaluated on their own when they are compiled. It leaves no room for structural / match by name typing.
There are a couple of approaches you could take here. The more laborous approach is to create another interface for the GetCount and Reset functions. Then create wrapper types for the two interfaces to plug into these methods. This is a bit tedious though after a while.
My personal preference is to solve this problem with delegates.
public Helper(TE source, Func<TE,count> getCount, Action<TE> reset)
{
_iface = source;
uint count = getCount(_iface);
reset(_iface);
...
}