I am using durable functions to wait for external events with timeouts. Even though one of the events is received before the timeout, a TimerFired event is recognised in dfMon when none should be.
The orchestration's logic is as follows:
orchstration will put a message on a queue which is monitored by an external system
this remote system will notify my orchestration via cmdReceived external event if the above queue message was received
then it will trigger a long-running local process
afterwards it will again notify my orchestration via cmdExecuted external event of the completion of this process
I.e. ->
PUT a message on the queue
timer1: wait 10mins for external event cmdReceived, continue if received, else throw
timer2: wait 60mins for external event cmdExecuted, continue if received, else throw
Now the cmdReceived event is received rather quickly, after 2mins or so. The the next timer should wait for a max of 60mins for cmdExecuted to be received. This usually takes about 12mins or so. By then 10mins (from timer1) have elapsed. And a TimerFired event is logged, however no exception is thrown and the orchestration keeps on running. But I do ask myself whether there is still an impact I am not immediately aware of. The existence of the TimerFired event (timer1) I mean.
This is the relevant line of code:
await ctx.WaitForExternalEvent("cmdReceived", TimeSpan.FromMinutes(10));
var success = await ctx.WaitForExternalEvent<bool>("cmdExecuted", TimeSpan.FromMinutes(60));
My first thought was, that maybe I need to explicitly cancel the timer myself. I've read about this here. So then I tried:
var cts = new CancellationTokenSource();
await ctx.WaitForExternalEvent("cmdReceived", TimeSpan.FromMinutes(10), cts.Token);
cts.Cancel();
var cts2 = new CancellationTokenSource();
var success = await ctx.WaitForExternalEvent<bool>("cmdExecuted", TimeSpan.FromMinutes(60), cts2.Token);
cts2.Cancel();
However, this didn't really help. The first timer keeps on firing (though not throwing) after 10mins even after the event for which it was configured has already been received.
See below for a screenshot from dfMon.
Is there anything wrong about this? I find this extra TimerFired event confusing. Does it matter as long as it is (correctly) not throwing an exception?
Cheers
Short answer: no it does not matter. The event is ignored by Durable Task.
The way that durable timers work with the default Storage durability provider is through scheduled queue messages.
So when you call WaitForExternalEvent, it uses CreateTimer under the hood.
When the replay finishes and Durable Task sees that there is a timer that needs to be started, it sends a scheduled message to one of the control queues.
This message will become visible at the time you specified.
Now when your orchestration receives the external event, that causes a replay, and it'll replay over the WaitForExternalEvent call.
Here it sees it has a new event for a received event.
This resolves the Task it is waiting on and your orchestration continues.
The next time it replays the timer event arrives.
But the timer gets ignored by processing the external event first.
(Internally it uses TaskCompletionSource.TrySetException() which won't do anything because SetResult has already been called)
It won't throw the timeout exception.
Regardless that it did nothing, the event did occur so it is recorded.
Related
Please bear with me as I'm very new to using Azure services. I'm using a service bus topic to read messages, and I have a RegisterMessageHandler for processing. I have been having problems on and off with the handler actually calling the function I pass to it. My code looks like this:
private void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
_subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private async Task ProcessMessagesAsync(Message message, CancellationToken cancellationToken)
{
// processing logic here
}
When I put a breakpoint in RegisterOnMessageHandlerAndReceiveMessages and in ProcessMessagesAsync, I can see that the line with _subscriptionClient is hit, but pressing Continue in debug mode doesn't jump into ProcessMessagesAsync. This was working a couple days ago, and the code I changed in the short time between then and now has nothing to do with either of these methods. What could be causing this?
Someone helped me figure it out! I just needed to delay for longer after calling RegisterMessageHandler, since it is lazily instantiated, so adding Thread.Sleep(50000) pretty much fixed it.
The problem with handler RegisterMessageHandler call is that it is not awaited for full completion. while code is running, main thread calls the handler RegisterMessageHandler event and moves to the next statement, which actually it should not have; and eventually reaches end of code, thereby forcing the called delegate (in this case "RegisterMessageHandler" method) to abort. The solution I think is to put a timer based waited approach or some kind of polling mechanism to check if handler is running or not
I have an async Method named "RequestMessage()". Within this method, I'm going to send a message to a message broker. Since I don't know when to expect the result, I'm using "TaskCompletionSource". I want the async method to terminate, when the reply message arrived (I'll receive an event from the broker).
This works fine so far. Now, my issue is that this message could never be answered, or at least way to late.
I'm looking for a change to implement my own timeout. To do so, I tried a Timer, as well as the Observer of Reactive Extensions. Th issue is always the same - I can't get my main thread and the timer thread syncronized, as I'm using .NET Core 2 and there is no SynchronizationContext.
So, in my code there is an observer ..
Observable
.Interval(TimeSpan.FromSeconds(timeOutInSeconds))
.Subscribe(
x =>
{
timeoutCallback();
});
If time expires a callback should be called. In my caller method, I handle the callback this way:
TimeoutDelegate timeoutHandler = () => throw new WorkcenterRepositoryCommunicationException("Broker communication timed out.", null);
As you realized already, this Exception will never be catched, as it is not thrown to the main thread.
How can I sync threads here?
Thanks in advance!
The best way to "fail upon some problem" IMHO would be to throw the appropriate exception, but you can definitely just use return; if you prefer to avoid exceptions.
This will create a completed/faulted task that was completed synchronously, so the caller using await will get a finished task and continue on using the same thread.
CancellationToken allows for the caller to cancel the operation, which isn't the case you are describing.
Task.Yield doesn't terminate any operation, it just enables other tasks to run for some time and reschedules itself for later.
I have a thread that grabs messages from a concurrent queue and writes them to a network stream. The loop inside the thread looks like this:
while (!cancel.IsCancellationRequested)
{
messageBus.outboundPending.WaitOne();
var message = messageBus.GetFrom(Direction.Outbound);
if (message == null) { continue; }
MessageWriter.WriteMessage(networkStream, message, cancel, OnStreamClose).Wait(cancel);
}
The requirement is that the thread stops if the cancellation token is set. However, since it waits for pending messages in the queue, the thread will stay blocked. How could I "combine" both the cancellation token and the outbound event so that if either of them are set, the thread unblocks?
The only convoluted way that I can think of to make it work is to replace the outboundPending event with a new third event, and start two new threads: one that waits for the outbound event, and another that waits for the cancel event, and have both of them set the third event when they unblock. But that feels really bad.
Try WaitHandle.WaitAny and include the CancellationToken.WaitHandle.
A discussion of a cancellable WaitAll can be found here
Use the WaitOne(TimeSpan) method. It will return true if it was signaled, and false if the timeout was reached.
e.g, if you send TimeSpan.FromSeconds(1) and a second has passed without a signal, the execution will continue and the method will return false. If the signal was given, it will return true.
I have a managed component written in C#, which is hosted by a legacy Win32 app as an ActiveX control. Inside my component, I need to be able to get what normally would be Application.Idle event, i.e. obtain a time slice of the idle processing time on the UI thread (it has to be the main UI thread).
However in this hosted scenario, Application.Idle doesn't get fired, because there is no managed message loop (i.e., no Application.Run).
Sadly, the host also doesn't implement IMsoComponentManager, which might be suitable for what I need. And a lengthy nested message loop (with Application.DoEvents) is not an option for many good reasons.
So far, the only solution I can think of is to use plain Win32 timers.
According to this (now perished) MSKB article, WM_TIMER has one of the lowest priorities, followed only by WM_PAINT, which should get me as close to the idle as possible.
Am I missing any other options for this scenario?
Here is a prototype code:
// Do the idle work in the async loop
while (true)
{
token.ThrowIfCancellationRequested();
// yield via a low-priority WM_TIMER message
await TimerYield(DELAY, token); // e.g., DELAY = 50ms
// check if there is a pending user input in Windows message queue
if (Win32.GetQueueStatus(Win32.QS_KEY | Win32.QS_MOUSE) >> 16 != 0)
continue;
// do the next piece of the idle work on the UI thread
// ...
}
// ...
static async Task TimerYield(int delay, CancellationToken token)
{
// All input messages are processed before WM_TIMER and WM_PAINT messages.
// System.Windows.Forms.Timer uses WM_TIMER
// This could be further improved to re-use the timer object
var tcs = new TaskCompletionSource<bool>();
using (var timer = new System.Windows.Forms.Timer())
using (token.Register(() => tcs.TrySetCanceled(), useSynchronizationContext: true))
{
timer.Interval = delay;
timer.Tick += (s, e) => tcs.TrySetResult(true);
timer.Enabled = true;
await tcs.Task;
timer.Enabled = false;
}
}
I don't think Task.Delay would be suitable for this approach, as it uses Kernel timer objects, which are independent of the message loop and its priorities.
Updated, I found one more option: WH_FOREGROUNDIDLE/ForegroundIdleProc. Looks exactly like what I need.
Updated, I also found that a Win32 timer trick is used by WPF for low-priority Dispatcher operations, i.e. Dispatcher.BeginInvoke(DispatcherPriority.Background, ...):
Well, WH_FOREGROUNDIDLE/ForegroundIdleProc hook is great. It behaves in a very similar way to Application.Idle: the hook gets called when the thread's message queue is empty, and the underlying message loop's GetMessage call is about to enter the blocking wait state.
However, I've overlooked one important thing. As it turns, the host app I'm dealing with has its own timers, and its UI thread is pumping WM_TIMER messages constantly and quite frequently. I could have learnt that if I looked at it with Spy++, in the first place.
For ForegroundIdleProc (and for Application.Idle, for that matter), WM_TIMER is no different from any other message. The hook gets called after each new WM_TIMER has been dispatched and the queue has become empty again. That results in ForegroundIdleProc being called much more often than I really need.
Anyway, despite the alien timer messages, the ForegroundIdleProc callback still indicates there is no more user input messages in the thread's queue (i.e., keyboard and mouse are idle). Thus, I can start my idle work upon it and implement some throttling logic using async/await, to kept the UI responsive. This is how it would be different from my initial timer-based approach.
I've got a MessageReceiver which is pumping messages from a queue:
var factory = MessagingFactory.CreateFromConnectionString(connectionString);
var receiver = factory.CreateMessageReceiver(queuePath);
receiver.OnMessageAsync(HandleBrokeredMessageAsync);
HandleBrokeredMessageAsync is my delegate which the receiver pumps messages into.
When I call Close() on the receiver, it will stop pumping further messages from the queue. In order to avoid potential race conditions, I want to be sure all pending processing has completed before returning control.
I have considered tracking each call to HandleBrokeredMessageAsync into a ConcurrentBag<T>, removing them from the bag when they complete. I'd use a BlockingCollection<T> to block the process until the drain-down is finished, but it's not clear when to call CompleteAdding(): I would call it after calling Close() but can there be a gap between calling Close() and a message being subsequently delivered to the handler?
receiver.Close();
pendingMessages.CompleteAdding();
// Can additional messages be pumped after this?
Take a look at the sample here as it uses a ManualResetEvent to co-ordinate the close of the Run() method and the Close operation. A similar approach may work where you check within message processing and stop there (not accept the next message for processing) and then call close when all those concurrent processors are done?
https://stackoverflow.com/a/16739691/1078351