We Keep Coding

Invoke inside Task.Run, how to solve deadlock?

I have a static method, which can be called from anywhere. During execution it will encounter Invoke. Obviously when this method is called from UI thread it will deadlock.
Here is a repro:
public static string Test(string text)
{
return Task.Run(() =>
{
App.Current.Dispatcher.Invoke(() => { } );
return text + text;
}).Result;
}
void Button_Click(object sender, RoutedEventArgs e) => Test();
I've read multiple questions and like 10 answers of #StephenCleary (even some blogs linked from those), yet I fail to understand how to achieve following:
have a static method, which is easy to call and obtain result from anywhere (e.g. UI event handlers, tasks);
this method should block the caller and after it the caller code should continue run in the same context;
this method shouldn't freeze UI.
The closest analogy to what Test() should behave like is MessageBox.Show().
Is it achieve-able?
P.S.: to keep question short I am not attaching my various async/await attempts as well as one working for UI calls, but terrible looking using DoEvents one.

You can not.
Even just 2 of those 3 requirements can't be achieved together - "this method should block the caller" is in conflict with "this method shouldn't freeze UI".
You have to make this method either asynchronous in some way (await, callback) or make it executable in small chunks to block UI only for short periods of time using for example timer to schedule each step.
Just to reiterate what you already know - you can't block thread and call it back at the same time as discusses in many questions like - await works but calling task.Result hangs/deadlocks.

To achieve something what MessageBox does (but without creating window) one can do something like this:
public class Data
{
public object Lock { get; } = new object();
public bool IsFinished { get; set; }
}
public static bool Test(string text)
{
var data = new Data();
Task.Run(() =>
{
Thread.Sleep(1000); // simulate work
App.Current.Dispatcher.Invoke(() => { });
lock (data.Lock)
{
data.IsFinished = true;
Monitor.Pulse(data.Lock); // wake up
}
});
if (App.Current.Dispatcher.CheckAccess())
while (!data.IsFinished)
DoEvents();
else
lock (data.Lock)
Monitor.Wait(data.Lock);
return false;
}
static void DoEvents() // for wpf
{
var frame = new DispatcherFrame();
Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background, new Func<object, object>(o =>
{
((DispatcherFrame)o).Continue = false;
return null;
}), frame);
Dispatcher.PushFrame(frame);
}
The idea is simple: check if current thread need invoke (UI thread) and then either run DoEvents loop or block thread.
Test() can be called from UI thread or from another task.
It works (not fully tested though), but it's crappy. I hope this will make my requirements clear and I still need the answer to my question if there is any better "no, you can't do this" ;)

Updating my ObservableCollection on a worker thread still hangs my UI

I have a log window in my application, when I have a few thousand logs, filtering them to include or exclude different log levels makes the UI unresponsive for a period of time from the work load. So I have tried to move the heavy lifting to a worker thread, and am having the same issue still.
I am using an ObservableCollection to hold the log information in my model, and I just reference that directly with my ViewModel. I have used BindingOperations.EnableCollectionSynchronization() to let my worker thread update my observable collection without Dispatching it to the UI thread.
I run the following to update the collection on a worker thread with a
Task.Run(new Action(() => FilterList()));
Methods:
private void FilterList()
{
//Necessary even with EnableCollectionSynchronization
App.Current.Dispatcher.Invoke(new Action(() =>
{
FilteredLogEvents.Clear();
}));
foreach (LogEvent log in FilterLogEvents())
{
FilteredLogEvents.Add(log);
}
RaisePropertyChanged("FilteredLogEvents");
FinishedFilteringLogs();
}
//Filters and returns a list of filtered log events
private List<LogEvent> FilterLogEvents()
{
List<LogEvent> selectedEvents = (from x in LogEvents
where ((ViewDebugLogs == true) ? x.Level == "Debug" : false)
|| ((ViewErrorLogs == true) ? x.Level == "Error" : false)
|| ((ViewInfoLogs == true) ? x.Level == "Info" : false)
select x).ToList();
return selectedEvents;
}
This causes the UI to freeze on the foreach. I also tried just newing up an ObservableCollection and then assigning FilteredLogEvents to it with FilteredLogEvents = myNewCollection; this also causes the UI to freeze for a short while during that process.
If I use Thread.Sleep(1) within the foreach loop the UI remains responsive, though this seems like an inelegant and hacky solution.
What do I need to do to make this work?
Edit: A bit more code context from this class (LogEntries)
The callback for FinishedFilteringLogsEventHandler goes back to the ViewModel to change a bool that enables a couple checkboxes when the filtering is complete.
//Constructor
public LogEntries()
{
foreach(NlogViewerTarget target in NLog.LogManager.Configuration.AllTargets.Where(t=>t is NlogViewerTarget).Cast<NlogViewerTarget>())
{
target.RecieveLog += RecieveLog;
}
FilteredLogEvents = new ObservableCollection<LogEvent>();
BindingOperations.EnableCollectionSynchronization(FilteredLogEvents, filteredLogEventsLock);
}
public delegate void FinishedFilteringLogsEvent();
public FinishedFilteringLogsEvent FinishedFilteringLogsEventHandler;
private object filteredLogEventsLock = new object();
public ObservableCollection<LogEvent> FilteredLogEvents { get; set; }

Some thoughts to consider to improve the speed and responsiveness of your code
A few days ago I asked a similar question and someone advised me not to use the threadpool for long running Tasks. The thread pool is a collection of available threads, that can be started swiftly in comparison to starting a traditional thread like System.ComponentModel.BackGroundWorker.
Although it takes more time to create and start a real thread, this is no problem for long running tasks.
The number of threads in the thread pool is limited, so better not use it for longer running tasks.
If you run a task, it is only scheduled to run in the near future when a thread is available. If all threads are busy it will take some time before the thread starts.
The change from Task to Backgroundworker is limited. If you really want to stick to tasks, consider creating an async function:
async void FilteredLogEvents.AddRangeAsync(IEnumerable<LogEvent> logEvents)
or maybe better:
async void FilteredLogEvents.SetAsync(IEnumerable<LogEvent> logEvents)
which does the clear and add in one async call.
Make your own function async:
private async void FilterList()
{
var filteredLogEvents = FilterLogEvents();
var myTask = Task.Run( () => FilteredLogEvents.SetAsync(filteredLogEvents);
// if desired do other things.
// wait until ready:
await myTask();
RaisePropertyChanged("FilteredLogEvents");
FinishedFilteringLogs();
}
By the way: Are you sure that your sequence of logEvents does not change while you are filtering it?
If so, why do you use ToList() instead of returning an IEnumerable and use deferred execution?
If you are not certain: what happens if during the FilterLogEvents your sequence of logEvents changes?

Is there such a synchronization tool as "single-item-sized async task buffer"?

Many times in UI development I handle events in such a way that when an event first comes - I immediately start processing, but if there is one processing operation in progress - I wait for it to complete before I process another event. If more than one event occurs before the operation completes - I only process the most recent one.
The way I typically do that my process method has a loop and in my event handler I check a field that indicates if I am currently processing something and if I am - I put my current event arguments in another field that is basically a one item sized buffer and when current processing pass completes - I check if there is some other event to process and I loop until I am done.
Now this seems a bit too repetitive and possibly not the most elegant way to do it, though it seems to otherwise work fine for me. I have two questions then:
Does what I need to do have a name?
Is there some reusable synchronization type out there that could do that for me?
I'm thinking of adding something to the set of async coordination primitives by Stephen Toub that I included in my toolkit.

So first, we'll handle the case that you described in which the method is always used from the UI thread, or some other synchronization context. The Run method can itself be async to handle all of the marshaling through the synchronization context for us.
If we're running we just set the next stored action. If we're not, then we indicate that we're now running, await the action, and then continue to await the next action until there is no next action. We ensure that whenever we're done we indicate that we're done running:
public class EventThrottler
{
private Func<Task> next = null;
private bool isRunning = false;
public async void Run(Func<Task> action)
{
if (isRunning)
next = action;
else
{
isRunning = true;
try
{
await action();
while (next != null)
{
var nextCopy = next;
next = null;
await nextCopy();
}
}
finally
{
isRunning = false;
}
}
}
private static Lazy<EventThrottler> defaultInstance =
new Lazy<EventThrottler>(() => new EventThrottler());
public static EventThrottler Default
{
get { return defaultInstance.Value; }
}
}
Because the class is, at least generally, going to be used exclusively from the UI thread there will generally need to be only one, so I added a convenience property of a default instance, but since it may still make sense for there to be more than one in a program, I didn't make it a singleton.
Run accepts a Func<Task> with the idea that it would generally be an async lambda. It might look like:
public class Foo
{
public void SomeEventHandler(object sender, EventArgs args)
{
EventThrottler.Default.Run(async () =>
{
await Task.Delay(1000);
//do other stuff
});
}
}
Okay, so, just to be verbose, here is a version that handles the case where the event handlers are called from different threads. I know you said that you assume they're all called from the UI thread, but I generalized it a bit. This means locking over all access to instance fields of the type in a lock block, but not actually executing the function inside of a lock block. That last part is important not just for performance, to ensure we're not blocking items from just setting the next field, but also to avoid issues with that action also calling run, so that it doesn't need to deal with re-entrancy issues or potential deadlocks. This pattern, of doing stuff in a lock block and then responding based on conditions determined in the lock means setting local variables to indicate what should be done after the lock ends.
public class EventThrottlerMultiThreaded
{
private object key = new object();
private Func<Task> next = null;
private bool isRunning = false;
public void Run(Func<Task> action)
{
bool shouldStartRunning = false;
lock (key)
{
if (isRunning)
next = action;
else
{
isRunning = true;
shouldStartRunning = true;
}
}
Action<Task> continuation = null;
continuation = task =>
{
Func<Task> nextCopy = null;
lock (key)
{
if (next != null)
{
nextCopy = next;
next = null;
}
else
{
isRunning = false;
}
}
if (nextCopy != null)
nextCopy().ContinueWith(continuation);
};
if (shouldStartRunning)
action().ContinueWith(continuation);
}
}

Does what I need to do have a name?
What you're describing sounds a bit like a trampoline combined with a collapsing queue. A trampoline is basically a loop that iteratively invokes thunk-returning functions. An example is the CurrentThreadScheduler in the Reactive Extensions. When an item is scheduled on a CurrentThreadScheduler, the work item is added to the scheduler's thread-local queue, after which one of the following things will happen:
If the trampoline is already running (i.e., the current thread is already processing the thread-local queue), then the Schedule() call returns immediately.
If the trampoline is not running (i.e., no work items are queued/running on the current thread), then the current thread begins processing the items in the thread-local queue until it is empty, at which point the call to Schedule() returns.
A collapsing queue accumulates items to be processed, with the added twist that if an equivalent item is already in the queue, then that item is simply replaced with the newer item (resulting in only the most recent of the equivalent items remaining in the queue, as opposed to both). The idea is to avoid processing stale/obsolete events. Consider a consumer of market data (e.g., stock ticks). If you receive several updates for a frequently traded security, then each update renders the earlier updates obsolete. There is likely no point in processing earlier ticks for the same security if a more recent tick has already arrived. Thus, a collapsing queue is appropriate.
In your scenario, you essentially have a trampoline processing a collapsing queue with for which all incoming events are considered equivalent. This results in an effective maximum queue size of 1, as every item added to a non-empty queue will result in the existing item being evicted.
Is there some reusable synchronization type out there that could do that for me?
I do not know of an existing solution that would serve your needs, but you could certainly create a generalized trampoline or event loop capable of supporting pluggable scheduling strategies. The default strategy could use a standard queue, while other strategies might use a priority queue or a collapsing queue.

What you're describing sounds very similar to how TPL Dataflow's BrodcastBlock behaves: it always remembers only the last item that you sent to it. If you combine it with ActionBlock that executes your action and has capacity only for the item currently being processed, you get what you want (the method needs a better name):
// returns send delegate
private static Action<T> CreateProcessor<T>(Action<T> executedAction)
{
var broadcastBlock = new BroadcastBlock<T>(null);
var actionBlock = new ActionBlock<T>(
executedAction, new ExecutionDataflowBlockOptions { BoundedCapacity = 1 });
broadcastBlock.LinkTo(actionBlock);
return item => broadcastBlock.Post(item);
}
Usage could be something like this:
var processor = CreateProcessor<int>(
i =>
{
Console.WriteLine(i);
Thread.Sleep(i);
});
processor(100);
processor(1);
processor(2);
Output:
100
2

Multi-threading problem when checking the list Count property

I have List newJobs. Some threads add items to that list and other thread removes items from it, if it's not empty. I have ManualResetEvent newJobEvent which is set when items are added to the list, and reset when items are removed from it:
Adding items to the list is performed in the following way:
lock(syncLock){
newJobs.Add(job);
}
newJobEvent.Set();
Jobs removal is performed in the following way:
if (newJobs.Count==0)
newJobEvent.WaitOne();
lock(syncLock){
job = newJobs.First();
newJobs.Remove(job);
/*do some processing*/
}
newJobEvent.Reset();
When the line
job=newJobs.First()
is executed I sometimes get an exception that the list is empty. I guess that the check:
if (newJobs.Count==0)
newJobEvent.WaitOne();
should also be in the lock statement but I'm afraid of deadlocks on the line newJobEvent.WaitOne();
How can I solve it?
Many thanks and sorry for the long post!

You are right. Calling WaitOne inside a lock could lead to a deadlock. And the check to see if the list is empty needs to be done inside the lock otherwise there could be a race with another thread trying to remove an item. Now, your code looks suspiciously like the producer-consumer pattern which is usually implemented with a blocking queue. If you are using .NET 4.0 then you can take advantage of the BlockingCollection class.
However, let me go over a couple of ways you can do it youself. The first uses a List and a ManualResetEvent to demonstrate how this could be done using the data structures in your question. Notice the use of a while loop in the Take method.
public class BlockingJobsCollection
{
private List<Job> m_List = new List<Job>();
private ManualResetEvent m_Signal = new ManualResetEvent(false);
public void Add(Job item)
{
lock (m_List)
{
m_List.Add(item);
m_Signal.Set();
}
}
public Job Take()
{
while (true)
{
lock (m_List)
{
if (m_List.Count > 0)
{
Job item = m_List.First();
m_List.Remove(item);
if (m_List.Count == 0)
{
m_Signal.Reset();
}
return item;
}
}
m_Signal.WaitOne();
}
}
}
But this not how I would do it. I would go with the simplier solution below with uses Monitor.Wait and Monitor.Pulse. Monitor.Wait is useful because it can be called inside a lock. In fact, it is suppose to be done that way.
public class BlockingJobsCollection
{
private Queue<Job> m_Queue = new Queue<Job>();
public void Add(Job item)
{
lock (m_Queue)
{
m_Queue.Enqueue(item);
Monitor.Pulse(m_Queue);
}
}
public Job Take()
{
lock (m_Queue)
{
while (m_Queue.Count == 0)
{
Monitor.Wait(m_Queue);
}
return m_Queue.Dequeue();
}
}
}

Not answering your question, but if you are using .NET framework 4, you can use the new ConcurrentQueue which does all the locking for you.
Regarding your question:
One scenario that I can think of causing such a problem is the following:
The insertion thread enters the lock, calls newJob.Add, leaves the lock.
Context switch to the removal thread. It checks for emptyness, sees an item, enters the locked area, removes the item, resets the event - which hasn't even been set yet.
Context switch back to the insertion thread, the event is set.
Context switch back to the removal thread. It checks for emptyness, sees no items, waits for the event - which is already set, trys to get the first item... Bang!
Set and reset the event inside the lock and you should be fine.

I don't see why object removal in case of zero objects should wait for one to be added and then remove it. It looks to be being against logic.

Reactive Framework (RX) and dealing with events Asynchronously

So I'm just playing around with RX and learning it. I started playing with Events, and wanted to know how to subscribe to events, and process the results in batches asynchronously. Allow me to explain with code:
Simple class that raises events:
public class EventRaisingClass
{
public event EventHandler<SomeEventArgs> EventOccured;
//some other code that raises event...
}
public class SomeEventArgs : EventArgs
{
public SomeEventArgs(int data)
{
this.SomeArg = data;
}
public int SomeArg { get; private set; }
}
Then my Main:
public static void Main(string[] args)
{
var eventRaiser = new EventRaisingClass();
IObservable<IEvent<SomeEventArgs>> observable =
Observable.FromEvent<SomeEventArgs>(e => eventRaiser.EventOccured += e, e => eventRaiser.EventOccured -= e);
IObservable<IList<IEvent<SomeEventArgs>>> bufferedEvents = observable.BufferWithCount(100);
//how can I subscribte to bufferedEvents so that the subscription code gets called Async?
bufferedEvents.Subscribe(list => /*do something with list of event args*/); //this happens synchrounously...
}
As you can see in my comments, when you just call subscribe like that, all the subscription code happens synchronously. Is there a way out of the box using RX to have the Subscribe be called on different threads whenever there's a new batch of events to work on?

bufferedEvents.ObserveOn(Scheduler.TaskPool).Subscribe(...
SubscribeOn is to specify the schedule on which so-called "subscription side effects" are happening. For example, your observable can open a file each time somebody subscribes.
ObserveOn is to specify the schedule on which the call to the observer will happen every time when there is a new value. In practice, it is used more often than SubscribeOn.

I believe you're looking for SubscribeOn or ObserveOn, passing an IScheduler. There are several schedulers built-in under System.Concurrency; some of them use whatever thread is current, and others use specific threads.
This video has more info on the scheduler concept.
The Rx team also recently released a hands-on labs document which is the closest thing to a tutorial right now.