I'm running this in a console application:
public void ForEachParallel(Action<TElement> action)
{
var elements = new Queue<TElement>(_set);
var tasks = Enumerable.Range(0, _threadCount)
.Where(index => elements.Any())
.Select(index => elements.Dequeue())
.Select(element => Task.Run(() => action(element)))
.ToList();
while (tasks.Any())
{
var index = Task.WaitAny(tasks.ToArray());
tasks.RemoveAt(index);
if (elements.Any())
{
var element = elements.Dequeue();
tasks.Add(Task.Run(() => action(element)));
}
}
}
I have an equivalent ForEach method that does all of this in a serial way. I'm using 10 threads, but the ForEachParallel is taking just as much time as the ForEach. I have an i7 with 6 cores. Either this has a whole lot of overhead, or it is somehow running these tasks with a single thread.
Each action is an independent read, process, and write.
Here's my test code:
void Main()
{
Action<int> action = n =>
{
Console.Write($" +{n} ");
Thread.Sleep(TimeSpan.FromSeconds(n + 1));
Console.Write($" {n}- ");
};
ForEachParallel(Enumerable.Range(0, 6), 4, action);
}
public void ForEachParallel<TElement>(IEnumerable<TElement> source, int threadCount, Action<TElement> action)
{
var elements = new Queue<TElement>(source);
var tasks =
source
.Take(threadCount)
.Where(index => elements.Any())
.Select(index => elements.Dequeue())
.Select(element => Task.Run(() => action(element)))
.ToList();
while (tasks.Any())
{
var index = Task.WaitAny(tasks.ToArray());
tasks.RemoveAt(index);
if (elements.Any())
{
var element = elements.Dequeue();
tasks.Add(Task.Run(() => action(element)));
}
}
}
It's a effectively the same as your ForEachParallel but I've made it more generic.
When I change the threadCount I get differing execution lengths. This is clearly running as expected.
Related
I have a LINQ query that should NOT be enumerated more than once, and I want to avoid enumerating it twice by mistake. Is there any extension method I can use to ensure that I am protected from such a mistake? I am thinking about something like this:
var numbers = Enumerable.Range(1, 10).OnlyOnce();
Console.WriteLine(numbers.Count()); // shows 10
Console.WriteLine(numbers.Count()); // throws InvalidOperationException: The query cannot be enumerated more than once.
The reason I want this functionality is because I have an enumerable of tasks, that is intended to instantiate and run the tasks progressivelly, while it is enumerated slowly under control. I already made the mistake to run the tasks twice because I forgot that it's a differed enumerable and not
an array.
var tasks = Enumerable.Range(1, 10).Select(n => Task.Run(() => Console.WriteLine(n)));
Task.WaitAll(tasks.ToArray()); // Lets wait for the tasks to finish...
Console.WriteLine(String.Join(", ", tasks.Select(t => t.Id))); // Lets see the completed task IDs...
// Oups! A new set of tasks started running!
I want to avoid enumerating it twice by mistake.
You can wrap the collection with a collection that throws if it's enumerated twice.
eg:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApp8
{
public static class EnumExtension
{
class OnceEnumerable<T> : IEnumerable<T>
{
IEnumerable<T> col;
bool hasBeenEnumerated = false;
public OnceEnumerable(IEnumerable<T> col)
{
this.col = col;
}
public IEnumerator<T> GetEnumerator()
{
if (hasBeenEnumerated)
{
throw new InvalidOperationException("This collection has already been enumerated.");
}
this.hasBeenEnumerated = true;
return col.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
public static IEnumerable<T> OnlyOnce<T>(this IEnumerable<T> col)
{
return new OnceEnumerable<T>(col);
}
}
class Program
{
static void Main(string[] args)
{
var col = Enumerable.Range(1, 10).OnlyOnce();
var colCount = col.Count(); //first enumeration
foreach (var c in col) //second enumeration
{
Console.WriteLine(c);
}
}
}
}
Enumerables enumerate, end of story. You just need to call ToList, or ToArray
// this will enumerate and start the tasks
var tasks = Enumerable.Range(1, 10)
.Select(n => Task.Run(() => Console.WriteLine(n)))
.ToList();
// wait for them all to finish
Task.WaitAll(tasks.ToArray());
Console.WriteLine(String.Join(", ", tasks.Select(t => t.Id)));
Hrm if you want parallelism
Parallel.For(0, 100, index => Console.WriteLine(index) );
or if you are using async and await pattern
public static async Task DoWorkLoads(IEnumerable <Something> results)
{
var options = new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 50
};
var block = new ActionBlock<Something>(MyMethodAsync, options);
foreach (var result in results)
block.Post(result);
block.Complete();
await block.Completion;
}
...
public async Task MyMethodAsync(Something result)
{
await SomethingAsync(result);
}
Update, Since you are after a way to control the max degree of conncurrency, you could use this
public static async Task<IEnumerable<Task>> ExecuteInParallel<T>(this IEnumerable<T> collection,Func<T, Task> callback,int degreeOfParallelism)
{
var queue = new ConcurrentQueue<T>(collection);
var tasks = Enumerable.Range(0, degreeOfParallelism)
.Select(async _ =>
{
while (queue.TryDequeue(out var item))
await callback(item);
})
.ToArray();
await Task.WhenAll(tasks);
return tasks;
}
Rx certainly is an option to control parallelism.
var query =
Observable
.Range(1, 10)
.Select(n => Observable.FromAsync(() => Task.Run(() => new { Id = n })));
var tasks = query.Merge(maxConcurrent: 3).ToArray().Wait();
Console.WriteLine(String.Join(", ", tasks.Select(t => t.Id)));
The below code will run for all sequences, I want the Observable to stop raising events after the condition is true.
private IObservable<string> EnsureIndexWithCounter(string index)
{
return Observable.Range(0, 5)
.SelectMany(p => IncrementCounterIfIndexExistsAsync(index, p)
.ToObservable()
.RepeatUntil(x => !x.Item1, 5))
.TakeWhile(p => !p.Item1)
.Select(p => p.Item2);
}
// Will be invoked 4 times, should be invoked as long the Item1 of the return tuple is true
private async Task<Tuple<bool, string>> IncrementCounterIfIndexExistsAsync(string index, int counter)
{
var existsResponse = await Client.IndexExistsAsync(new IndexExistsRequest(index)).ConfigureAwait(false);
var newCounter = existsResponse.Exists ? ++counter : counter;
return Tuple.Create(existsResponse.Exists, $"{index}_{newCounter}");
}
I have this code from old examples (Reactive UI 4):
StartAsyncCommand = new ReactiveCommand();
StartAsyncCommand.RegisterAsyncAction(_ =>
{
Progress = 0;
var exe = Enumerable.Range(0, 10).Select(x =>
{
Thread.Sleep(100);
return x;
}).ToObservable();
exe.Subscribe(x =>Progress += 10);
});
It works fine, UI does not block and button is disable until Progress gets 100%.
When I migrate to version 6, I tried many ways to achieve same functionality without success.
These are my attempts:
1) Using CreateAsyncObservable
GenerateCommand = ReactiveCommand.CreateAsyncObservable(canGenerate, x => DoSomething());
public IObservable<Unit> DoSomething()
{
var exe = Enumerable.Range(0, 10).Select(
x =>
{
Thread.Sleep(200);
return x;
}).ToObservable().ObserveOnDispatcher().SubscribeOn(NewThreadScheduler.Default);
exe.Subscribe(x =>
Progress += 10
);
return Observable.Return(new Unit());
}
It works but button does not disable.
2) With CreateAsyncTask
GenerateCommand = ReactiveCommand.CreateAsyncTask(canGenerate, x => DoSomething());
public async Task<Unit> DoSomething()
{
var exe = Enumerable.Range(0, 10).Select(
x =>
{
Thread.Sleep(200);
return x;
}).ToObservable().ObserveOnDispatcher().SubscribeOn(NewThreadScheduler.Default);
await exe;
exe.Subscribe(x =>
Progress+=10
);
return new Unit();
}
It works but button only disable until await ends.
3) Based on ReactiveUI 6 Async Command Not Running on Background Thread in WPF app
GenerateCommand = ReactiveCommand.CreateAsyncTask(canGenerate, x => DoSomething());
public async Task<Unit> DoSomething()
{
var execTask = Task.Factory.StartNew(() =>
{
var exe = Enumerable.Range(0, 10).Select(
x =>
{
Thread.Sleep(200);
return x;
}).ToObservable();
exe.Subscribe(x =>
Progress += 10
);
return new Unit();
});
return execTask.Result;
}
This throws an exception.
UPDATE
4) Subscribe command:
GenerateCommand = ReactiveCommand.CreateAsyncObservable(canGenerate, x => DoSomething());
GenerateCommand.ObserveOnDispatcher().SubscribeOn(NewThreadScheduler.Default)
.Subscribe(x => Progress += 10);
public IObservable<int> DoSomething()
{
return = Observable.Range(0, 10).Select(
x =>
{
Thread.Sleep(200);
return x;
});
}
Same result...
How could I achieve same functionality with version 6?
What am I doing wrong?
This is a very odd way to do what you're trying to do. How about this instead:
StartAsyncCommand = ReactiveCommand.CreateAsyncObservable(_ =>
Observable.Timer(DateTimeOffset.Zero, TimeSpan.FromSeconds(1))
.Take(10)
.Scan(0, (acc,x) => acc + 10));
StartAsyncCommand.ToProperty(this, x => x.Progress, out progress);
I have a little problem with Threads in this code..
I just want to run a lot of tasks together, and continue when all of them finish.
while (true)
{
// Run tasks together:
foreach (object T in objectsList)
{
if (T.something>0)
var task = Task.Factory.StartNew(() => T.RunObject());
task.ContinueWith(delegate { ChangeObject(T, 1); }, TaskContinuationOptions.NotOnFaulted);
}
// <-- Here I want to wait for all the task to be finish.
// I know its task.Wait() but how to waitAll()?
System.Threading.Thread.Sleep(this.GetNextTime());
var RefreshObjects = new Task(loadObjectsList); RefreshObjects .Start(); RefreshObjects.Wait();
}
I don't know how many objects will be in objectsList and I don't know if T.something will be > 0.
so I can't just use:
Task[] Tasks = new Task[objectsList.count()]
for (int T=0; T<objectsList.count(); ++T)
{
if (objectsList[T].something>0)
var task = Task.Factory.StartNew(() => objectsList[T].RunObject());
task.ContinueWith(delegate { ChangeObject(objectsList[T], 1); }, ...);
}
Task.WaitAll(Tasks);
Because Tasks will contains nulls when objectsList[T].something!>0...
Thanks for any advice!
Just switch the condition and create a List of tasks only for the objects which matches your criteria.
var tasks = objectsList
.Where(x => x.Something() > 0)
.Select(x => {
var task = Task.Factory.StartNew(() => x.RunObject());
task.ContinueWith(t => ChangeObject(....));
return task;
})
.ToArray();
Task.WaitAll(tasks);
Your code sample just waits for RunObject()to complete! If this is desired skip the rest of my answer. If you want to wait for the continuation to complete, too you can use this
var tasks = objectsList
.Where(x => x.Something() > 0)
.Select(x => Task.Factory.StartNew(() => x.RunObject()).ContinueWith(t => ChangeObject(....)))
.ToArray();
Task.WaitAll(tasks);
because ContinueWith generates a new Task.
If objectsList implements IEnumerable, (as an array does),
(And there are less than 64 objects in the list), you can use this:
public delegate void SyncDelegatesInParallelDelegate<in T>(T item);
public static class ParallelLinqExtensions
{
public static void SyncDelegatesInParallel<T>(
this IEnumerable<T> list,
SyncDelegatesInParallelDelegate<T> action)
{
var foundCriticalException = false;
Exception exception = null;
var waitHndls = new List<WaitHandle>();
foreach (var item in list)
{
// Temp copy of session for modified closure
var localItem = item;
var txEvnt = new ManualResetEvent(false);
// Temp copy of session for closure
ThreadPool.QueueUserWorkItem(
depTx =>
{
try { if (!foundCriticalException) action(localItem); }
catch (Exception gX)
{ exception = gX; foundCriticalException = true; }
finally { txEvnt.Set(); }
}, null);
waitHndls.Add(txEvnt);
}
if (waitHndls.Count > 0) WaitHandle.WaitAll(waitHndls.ToArray());
if (exception != null) throw exception;
}
}
you would call it like this
objectsList.SyncDelegatesInParallel(delegate { ChangeObject(T, 1);});
I want to iterate a collection by moving from element to element during specific time intervals. So for example this method works fine:
var a = new List<int> { 1, 2, 3, 4}.ToObservable();
var b = Observable.Interval(TimeSpan.FromSeconds(1));
var c = a.Zip(b, (l, r) => l);
c.Subscribe(x => Debug.WriteLine(x));
But I would like to use the value of each element in the list as the interval, so I am using this code:
var a = new List<int> { 1, 2, 3, 4}.ToObservable();
var b = a.Delay(x => Observable.Timer(TimeSpan.FromSeconds(x)));
b.Subscribe(x => Debug.WriteLine(x));
As it stated here http://blogs.msdn.com/b/rxteam/archive/2012/03/12/reactive-extensions-v2-0-beta-available-now.aspx "the new overloads to Delay allow one to specify (optionally) the delay for the subscription as well as the delay for each element based on a selector function". But running the code does not work like expected. It just spits out the elements of the list using 1 sec interval. Like so:
...(1 sec)
1
...(1 sec)
2
...(1 sec)
3
...(1 sec)
4
Instead of
...(1 sec)
1
...(2 sec)
2
...(3 sec)
3
...(4 sec)
4
Am I missing something?
How about this:
new[] {1,2,3,4,5,}.ToObservable()
.Select(x => Observable.Return(x).Delay(TimeSpan.FromSeconds(x)))
.Concat();
Right? You want to use the number as both the value as well as the amount of time to delay?
Here is a solution that advances the iterator on the enumerable source only one step before it waits again (works with infinite enumerable source sequences):
static IObservable<T> ToObservableDelay<T>(
IEnumerable<T> source,
Func<T, TimeSpan> delaySelector,
IScheduler scheduler
)
{
return Observable.Create<T>(o =>
scheduler.ScheduleAsync(async (s, c) =>
{
try
{
foreach (var x in source)
{
await Task.Delay(delaySelector(x), c);
o.OnNext(x);
}
o.OnCompleted();
}
catch (TaskCanceledException) { /* ignore */ }
catch (Exception e)
{
o.OnError(e);
}
})
);
}
And here a little demo with an infinite generator:
static IEnumerable<double> Uniform(Random rng)
{
while (true)
yield return rng.NextDouble();
}
static void Main(string[] args)
{
var source = Uniform(new Random());
Console.WriteLine("press any key to quit");
using (var subscription =
ToObservableDelay(source, TimeSpan.FromSeconds, Scheduler.Default)
.Subscribe(Console.WriteLine))
{
Console.ReadKey();
}
}
Simple demo code that runs in a RT app:
var source = Uniform(new Random());
ToObservableDelay(source, TimeSpan.FromSeconds, Scheduler.Default)
.Take(10)
.Subscribe(x => Debug.WriteLine(x));
I found this question while trying to solve the same issue. My solution was to use Observable.Generator together with the IEnumerable's enumerator, like so:
public static IObservable<T> ToObservable<T>(this IEnumerable<T> source, TimeSpan time)
{
var enumerator = source.GetEnumerator();
var observable = Observable.Generate(
enumerator.Current,
_ => enumerator.MoveNext(),
_ => enumerator.Current,
_ => enumerator.Current,
_ => time);
return observable;
}
This extension method is then simply called like this:
var observable = Enumerable.Range(1, 10)
.ToObservable(TimeSpan.FromSeconds(1));
using (observable.Subscribe(Console.WriteLine))
{
Console.WriteLine("Press the Any key to abort");
Console.ReadKey();
}