We Keep Coding

How can I send a batch of messages to an Azure EventHub? (C# WinForm and .NET 5.0)

I am building a telemetry simulator to send messages to an Azure EventHub (WinForm, .NET 5.0).
I use backgroundworkers for each telemetry device, the code below is the DoWork method of one of the devices. The app works fine when I output the messages to the console.
The problem occurs when I add the (commented out) EventHub code shown in the while loop below. There are two issues:
Backgroundworker.ReportProgress fails with: System.InvalidOperationException: 'This operation has already had OperationCompleted called on it and further calls are illegal.'
I also can no longer cancel the process (i.e. truckWorker.CancellationPending is always false)
If I uncomment the EventHub code, and comment out truckWorker.ReportProgress(i++), the app works and sends messages to the EventHub. Problem is I still can't cancel the operation and of course I loose my progress indicator.
private async void SendTruckMsg(object sender, DoWorkEventArgs e)
{
EventHubProducerClient evClient = (EventHubProducerClient)e.Argument;
List<TruckTelemetry> collection = virtualThingsController.GetTruckTelemetry();
int interval = virtualThingsController.GetTruckTelemetryInterval();
int i = 0; // count messages sent
while (!truckWorker.CancellationPending) // ===> can't cancel operation anymore
{
//using var eventBatch = await evClient.CreateBatchAsync(); // create batch
foreach (TruckTelemetry truckTelemetry in collection)
{
truckTelemetry.Timestamp = DateTime.Now;
string output = JsonConvert.SerializeObject(truckTelemetry);
//eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(output))); // add to batch
}
//await evClient.SendAsync(eventBatch); // send the batch
truckWorker.ReportProgress(i++);
Thread.Sleep(interval);
}
}
Sending messages could be synchronous but there is no simple 'send' method for the EventHubProducerClient.
Appreciate any help, thanks.

Before I touch on the background processing, there's a couple of things that I'm seeing in your snippet that look like they're likely to be a problem in your application:
It appears that you're using a List<TruckTelemetry> on multiple threads, one that adds items and the background operation that publishes them to Event Hubs. The list isn't thread-safe. I'd recommend moving to a ConcurrentQueue<T>, which is thread-safe and will help preserve the ordering of your telemetry.
Your snippet is ignoring the return of the TryAdd method of the EventDataBatch and assuming that every telemetry item in your loop was successfully added to the batch. This can lead to data loss, as you would be unaware if the batch was full and events were not able to be added.
Your snippet doesn't handle the corner case where a telemetry item is too large and could never be published to Event Hubs. Granted, if your telemetry items are tiny, this isn't a case that you're likely to encounter, but I'd recommend including it just to be safe.
With asynchronous operations, the typical approach to working in the background is to start a new Task that you can cancel or wait to complete when it makes sense for the application.
If I translate your general scenario, turning it into a task and addressing my feedback, your telemetry processing would look something like:
public static Task ProcessTruckTelemetry(EventHubProducerClient producer,
VirtualThingsController controller,
CancellationToken cancellationToken,
Action<int> progressCallback) =>
Task.Run(async () =>
{
var eventBatch = default(EventDataBatch);
var totalEventCount = 0;
var interval = controller.GetTruckTelemetryInterval();
// I'm assuming a change to ConcurrentQueue<T> for the
// the telemetry interactions via the controller.
var telemetryQueue = controller.GetTruckTelemetry();
while (!cancellationToken.IsCancellationRequested)
{
// In this example, we'll pump the telemetry queue as long
// long as there are any items in there. If you wanted to
// limit, you could also include a maximum count and stop there.
while ((!cancellationToken.IsCancellationRequested)
&& (telemetryQueue.TryPeek(out var telemetry))
{
// Create a batch if we don't currently have one.
eventBatch ??= (await producer.CreateBatchAsync().ConfigureAwait(false));
// Translate the telemetry data.
telemetry.Timestamp = DateTime.UtcNow;
var serializedTelemetry = JsonSerializer.Serialize(telemetry);
var eventData = new EventData(new BinaryData(serializedTelemetry));
// Attempt to add the event to the batch. If the batch is full,
// send it and clear state so that we know to create a new one.
if (!eventBatch.TryAdd(eventData))
{
// If there are no events in the batch, this event is
// too large to ever publish. We can't recover.
//
// An important note in this scenario is that we have
// already removed the telemetry from the queue. If we don't
// want to lose that, we should take action before throwing the
// exception to capture it.
if (eventBatch.Count == 0)
{
throw new Exception("There was an event too large to publish.");
}
await producer.SendAsync(eventBatch).ConfigureAwait(false);
totalEventCount += eventBatch.Count;
eventBatch.Dispose();
eventBatch = default;
}
}
else
{
telemetryQueue.TryDequeue(out _);
}
// Once we hit this point, there were no telemetry items left
// in the queue. Send any that are held in the event batch.
if ((eventBatch != default) && (eventBatch.Count > 0))
{
await producer.SendAsync(eventBatch).ConfigureAwait(false);
totalEventCount += eventBatch.Count;
eventBatch.Dispose();
eventBatch = default;
}
// Invoke the progress callback with the total count.
progressCallback(totalEventCount);
// Pause for the requested delay before attempting to
// pump the telemetry queue again.
try
{
await Task.Delay(interval, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Thrown if cancellation is requested while we're in the
// delay. This example is assuming that it isn't interesting
// to the application and swallows it.
}
}
}, cancellationToken);
Which your application would interact with using something similar to:
// This is your master shutdown signal. When you request
// cancellation on this token, all of your background tasks
// should terminate.
using var cancellationSource = new CancellationTokenSource();
var producer = GetProducerClient();
var virtualThingsController = new VirtualThingsController();
try
{
// Define a simple process callback.
Action<int> progressCallback = totalCount =>
Debug.WriteLine($"There have been a total of { totalCount } items published.");
// Start the background processing and capture the tasks. Once the
// call is made, telemetry is being processed in the backgorund until
// the cancellation token is signaled.
var backgroundTasks = new List<Task>();
backgroundTasks.Add
(
ProcessTruckTelemetry(
producer,
virtualThingsController,
cancellationSource.Token,
progressCallback)
);
backgroundTasks.Add
(
ProcessOtherTelemetry(
producer,
virtualThingsController,
cancellationSource.Token,
progressCallback)
);
// The application can do whatever it normally does now.
//
// << STUFF >>
//
// When the application is ready to stop, signal the cancellation
// token and wait for the tasks. We're not calling ConfigureAwait(false)
// here because your application is WinForms, which has some
// sensitivity to the syncrhonization context.
cancellationSource.Cancel();
await Task.WhenAll(backgroundTasks);
}
finally
{
await producer.CloseAsync();
}

How can I properly extend the background time of an iOS app so that an HTTP request doesn't get cancelled?

I have an app (App1) that makes use of the WKWebView for a good portion of the UI. There is a scenario where an HTTP PUT request is sent from the WKWebView to a backend server to save some data. For this save operation to complete, the server will need approval thru another app (App2). The user would normally switch to App2 to approve, then switch back to App1 to see the result of the save. The problem is that when App1 gets backgrounded, it can cause the response to the save request to be cancelled, even though the save was completely successful on the backend server. There isn't any errors actually logged, but I'm fairly certain it is happening because iOS is killing the connection when the app gets suspended after it gets backgrounded. I'm basing my thoughts on this discussion.
Since the time it takes to approve the save on App2 isn't that long, I figured I could just try to extend the background time of App1, and it appears to work in the times I've tested it.
However, I want to know if this is really the best strategy, and if so, are there any recommendations on my code (For example, should I move the BeginBackgroundTask inside of the Task.Run):
I used these microsoft docs as an example.
public override async void DidEnterBackground(UIApplication application)
{
ExtendBackgroundTime(application);
}
private nint? webViewBgTaskId = null;
private CancellationTokenSource webViewBgTaskTokenSrc = null;
private void ExtendBackgroundTime(UIApplication application)
{
// cancel the previous background task that was created in this function
webViewBgTaskTokenSrc?.Cancel();
webViewBgTaskTokenSrc = null;
if (webViewBgTaskId.HasValue)
{
application.EndBackgroundTask(webViewBgTaskId.Value);
webViewBgTaskId = null;
}
var cts = new CancellationTokenSource();
nint taskId = default;
taskId = application.BeginBackgroundTask(() =>
{
cts.Cancel();
webViewBgTaskTokenSrc = null;
application.EndBackgroundTask(taskId);
webViewBgTaskId = null;
});
_ = Task.Run(async () =>
{
// For now, this is just set to 5 minutes, but in my experience,
// the background task will never be allowed to continue for that long.
// It's usually only about 30 seconds as of iOS 13.
// But this at least gives it some finite upper bound.
await Task.Delay(TimeSpan.FromMinutes(5), cts.Token);
application.EndBackgroundTask(taskId);
webViewBgTaskId = null;
}, cts.Token);
webViewBgTaskTokenSrc = cts;
webViewBgTaskId = taskId;
}

The following code snippet demonstrates registering a task to run in the background:
nint taskID = UIApplication.SharedApplication.BeginBackgroundTask( () => {});
//runs on main or background thread
FinishLongRunningTask(taskID);
UIApplication.SharedApplication.EndBackgroundTask(taskID);
The registration process pairs a task with a unique identifier, taskID, and then wraps it in matching BeginBackgroundTask and EndBackgroundTask calls. To generate the identifier, we make a call to the BeginBackgroundTask method on the UIApplication object, and then start the long-running task, usually on a new thread. When the task is complete, we call EndBackgroundTask and pass in the same identifier. This is important because iOS will terminate the application if a BeginBackgroundTask call does not have a matching EndBackgroundTask.
Note: If you want to perform Tasks During DidEnterBackground method, these tasks must be invoked on a separate thread. Therefore, sample project uses Task to invoke FinishLongRunningTask.
Task.Factory.StartNew(() => FinishLongRunningTask(taskID));

When a message fails to process in a ServiceBusTrigger Azure Function, how can I delay processing the same message for x minutes?

I have an azure function that reads from a ServiceBus topic and calls a 3rd party service. If the service is down, I would like to wait 5 minutes before trying to call it again with the same message. How can I add a delay so the azure function doesn't abandon the message and immediately pick it back up again?
public static void Run([ServiceBusTrigger("someTopic",
"someSubscription", AccessRights.Manage, Connection =
"ServiceBusConnection")] BrokeredMessage message)
{
CallService(bodyOfBrokeredMessage); //service is down
//How do I add a delay so the message won't be reprocessed immediately thus quickly exhausting it's max delivery count?
}

One option is to create a new message and submit that message to the queue but set the ScheduledEnqueueTimeUtc to be five minutes in the future.
[FunctionName("DelayMessage")]
public static async Task DelayMessage(
[ServiceBusTrigger("MyQueue", AccessRights.Listen, Connection = "MyConnection")]BrokeredMessage originalMessage,
[ServiceBus("MyQueue", AccessRights.Send, Connection = "MyConnection")]IAsyncCollector<BrokeredMessage> newMessages,
TraceWriter log)
{
//handle any kind of error scenerio
var newMessage = originalMessage.Clone();
newMessage.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(5);
await newMessages.AddAsync(newMessage);
}

You can now use the fixed delay retry, which was added to Azure Functions around November 2020 (preview).
[FunctionName("MyFunction")]
[FixedDelayRetry(10, "00:05:00")] // retries with a 5-minute delay
public static void Run([ServiceBusTrigger("someTopic",
"someSubscription", AccessRights.Manage, Connection =
"ServiceBusConnection")] BrokeredMessage message)
{
CallService(bodyOfBrokeredMessage); //service is down
}

As Josh said, you could simply clone the original message, set up the scheduled enqueue time, send the clone and complete the original.
Well, it’s a shame that sending the clone and completing the original are not an atomic operation, so there is a very slim chance of us seeing the original again should the handling process crash at just the wrong moment.
And the other issue is that DeliveryCount on the clone will always be 1, because this is a brand new message. So we could infinitely resubmit and never get round to dead-lettering this message.
Fortunately, that can be fixed by adding our own resubmit count as a property of the message:
[FunctionName("DelayMessage")]
public static async Task DelayMessage([ServiceBusTrigger("MyQueue", AccessRights.Listen, Connection = "MyConnection")]BrokeredMessage originalMessage,
[ServiceBus("MyQueue", AccessRights.Send, Connection = "MyConnection")]IAsyncCollector<BrokeredMessage> newMessages,TraceWriter log)
{
//handle any kind of error scenerio
int resubmitCount = originalMessage.Properties.ContainsKey("ResubmitCount") ? (int)originalMessage.Properties["ResubmitCount"] : 0;
if (resubmitCount > 5)
{
Console.WriteLine("DEAD-LETTERING");
originalMessage.DeadLetter("Too many retries", $"ResubmitCount is {resubmitCount}");
}
else
{
var newMessage = originalMessage.Clone();
newMessage.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(5);
await newMessages.AddAsync(newMessage);
}
}
For more details, you could refer to this article.
Also, it's quite easy to implement the wait/retry/dequeue next pattern in a LogicApp since this type of flow control is exactly what LogicApps was designed for. Please refer to this SO thread.

Azure Service Bus as Web Job throw error The lock supplied is invalid

Currently I'm tring to implement my service queue bus on web job. The process that i'm perform with each message is taking about 5 - 30 seconds. While I'm not getting many messages in same time it's running ok, without any exceptions. Otherwise I'm getting this error: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue.
I'm read something about time that I should use to avoid of this error, but it doesn't help me (I'm still getting this error) and I dont' know why it's happen? Maybebe someone stack on similiar problem and solve it with other solution that i use (I'm change MaxAutoRenewDuration to 5 minutes).
Maybe is something wrong with my web job implementation ?
Here's my code:
static void Main(string[] args)
{
MainAsync().GetAwaiter().GetResult();
}
static async Task MainAsync()
{
JobHostConfiguration config = new JobHostConfiguration();
config.Tracing.ConsoleLevel = System.Diagnostics.TraceLevel.Error;
queueClient = new QueueClient(ServiceBusConnectionString, QueueName);
RegisterOnMessageHandlerAndReceiveMessages();
JobHost host = new JobHost(config);
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
host.RunAndBlock();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
MaxAutoRenewDuration = TimeSpan.FromMinutes(5),
AutoComplete = false
};
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
var watch = System.Diagnostics.Stopwatch.StartNew();
Console.WriteLine("----------------------------------------------------");
try
{
Thread.Sleep(15000); // average time of actions that i perform
watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;
var results = true;
}
catch (Exception ex)
{
}
Console.WriteLine("----------------------------------------------------");
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}

MessageHandlerOptions has a ExceptionReceivedHandler callback you could use to get more details about the failure.
Losing lock can take place, especially if client fails to communicate back to the server on time or there are intermittent failures Azure Service Bus retries itself, but takes time. Normal LockDuration time is 60 seconds, so your sample code should have worked. It could be that you're experiencing connectiving issues that are retried by the client and by then lock is expired. Another option, clock skew between your local machine and the server, which speeds up lock expiration. You could sync the clock to eliminate that.
Note that MaxAutoRenewDuration is not as effective as LockDuration. It's better to set the LockDuration to the maximum that rely on MaxAutoRenewDuration.
In case this code is not what you've used to repro the issue, please share the details.

Hanging with WaitForConnectionAsync() on pipe [duplicate]

This question already has answers here:
await vs Task.Wait - Deadlock?
(3 answers)
Closed 7 years ago.
I'm communicating with another process via named pipes. The pipe server is implemented in C# and the client is written in C. The server is a WPF application.
I need to create a NamedPipeServerStream and wait (synchronously) up to 1 second for the client to connect. And then I need to know whether the client connected.
As NamedPipeServerStream's only way to cancel/timeout a wait for the client to connect is via its asynchronous WaitForConnectionAsync method - which takes a CancellationToken - I've implemented what I believe is a synchronous wait like so:
public bool WaitOneSecondForClientConnect()
{
bool result = false;
try
{
result = WaitForConnectionAsyncSyncWrapper().Result;
}
catch (AggregateException e)
{
log.Write("Error waiting for pipe client connect: " + e.InnerException.Message);
}
return result;
}
private async Task<bool> WaitForConnectionAsyncSyncWrapper()
{
CancellationTokenSource cts = new CancellationTokenSource(1000);
await pipe.WaitForConnectionAsync(cts.Token);
return pipe.IsConnected;
}
The pipe is defined like so: NamedPipeServerStream(pipeName, PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, 1, 1);
The WaitOneSecondForClientConnect() function runs on the UI thread.
What should make it synchronous is accessing the async WaitForConnectionAsyncSyncWrapper() function's Result property on the Task<bool> it returns. In order to access the Result, the async function must have fully returned, and it can't execute the return pipe.IsConnected line until the function is resumed after await pipe.WaitForConnectionAsync(cts.Token); has completed. At least that's my understanding.
So the problem: although the client program says it's opened my server's pipe (which has already been created of course before the code above executes), WaitOneSecondForClientConnect() never returns. If I break into the server, it's on this line: result = WaitForConnectionAsyncSyncWrapper().Result;.
So I guess it's waiting for the Task's Result to be available, which should be the value of pipe.IsConnected if the client connected within 1 second, or it should throw an AggregateException when I access it if the await has completed because the token has been cancelled (after 1 second). But it's just hanging completely.
On the other hand, if I cancel the token before it starts e.g. by putting a Thread.Sleep(2000); right before calling await pipe.WaitForConnectionAsync(cts.Token);, then the connection is cancelled successfully (I think it doesn't even try to start because the token is already cancelled) - accessing the Result property throws an AggregateException, etc...
A few things to note.
If I replace the content of WaitOneSecondForClientConnect() with a
standard synchronous pipe.WaitForConnection();, it works every time -
i.e. the client connects, and the function returns.
In a test program I wrote to initially get this async/sync stuff working, connecting in this synchronous-asynchronous way works every time. It's a console program as opposed to my real program which is WPF. The relevant code of the test program is listed below.
The code posted above has actually worked a couple of times, and failed maybe 30-40 times.
If my client doesn't open my pipe, my "real" code still hangs, whereas my test code waits the specified time period and then prints "Connection failed." as expected (see below) - which is exactly the behaviour that should be happening in my real code.
The test code that works:
var pipe = new NamedPipeServerStream("SemiUsefulPipe_" + pid.ToString() + "ctest", PipeDirection.Out, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, 1, 1);
// ... dll containing pipe client is injected in client process at this point.
try
{
var result = ConnectAsync(pipe).Result;
}
catch (AggregateException)
{
Console.WriteLine("Connection failed.");
}
...
private static async Task<bool> ConnectAsync(NamedPipeServerStream pipe)
{
CancellationTokenSource cts = new CancellationTokenSource(1000);
await pipe.WaitForConnectionAsync(cts.Token);
return pipe.IsConnected;
}

You can't mix async and non-async methods like this. What's happening is that your WaitOneSecondForClientConnect method is waiting for WaitForConnectionAsyncSyncWrapper method to complete. But that guy needs its calling thread to be free so it can rehydrate the original context. So you've just created a deadlock. For details, see https://msdn.microsoft.com/en-us/magazine/jj991977.aspx.
Instead, you need to go async all the way down.
public async Task<bool> WaitOneSecondForClientConnect()
{
bool result = false;
try
{
result = await WaitForConnectionAsyncSyncWrapper();
}
catch (Exception e)
{
log.Write("Error waiting for pipe client connect: " + e.Message);
}
return result;
}