We Keep Coding

TPL .Net Concurrenty Issue with Azure EventHub Producer

I am working with Azure Event Hub producer client and reading messages off of a kafka stream then pass it along to deserialize/map, then pass to Event Hub. I have the consume loop which is creating a task for each consume and then two methods to do processing(this seems to have greatly improved the speed from the kafka lag perspective. However, Event hub makes you create an event batch which I don't necessarily want to use. I just want to send the data one message at a time for now. In order to create a new batch I have to call Dispose(). I am running into an issue where there's another call to the function by the the time I call Dispose() and I get an error saying the object is being used by event hub.
I've also tried using the overload for eventHubProducerClient.SendAsync that allows you to pass in a IEnumerable but i'm running into the same issue with that.
So I believe this to be a synchronization issue, or maybe I need to do a lock somewhere?
Any help would be appreciated.
public void Execute()
{
using (_consumer)
{
try
{
_consumer.Subscribe(_streamConsumerSettings.Topic);
while (true)
{
var result = _consumer.Consume(1000);
if (result == null)
{
continue;
}
var process = Task.Factory.StartNew(() => ProcessMessage(result?.Message?.Value));
var send = process.ContinueWith(t => SendMessage(process.Result));
}
}
catch (ConsumeException e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
_cancelConsume = true;
_consumer.Close();
RestartConsumer();
}
}
}
public static EquipmentJson ProcessMessage(byte[] result)
{
var json = _messageProcessor.DeserializeAndMap(result);
return json;
}
public static void SendMessage(EquipmentJson message)
{
try
{
_eventHubClient.AddToBatch(message);
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
public async Task AddToBatch(EquipmentJson message)
{
if
(!string.IsNullOrEmpty(message.EquipmentLocation))
{
try
{
var batch = await _equipmentLocClient.CreateBatchAsync();
batch.TryAdd(new EventData(Encoding.UTF8.GetBytes(message.EquipmentLocation)));
await _eventHubProducerClient.SendAsync(batch);
batch.Dispose();
_logger.LogInformation($"Data sent {DateTimeOffset.UtcNow}");
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
}
public class EventHubClient : IEventHubClient
{
private readonly ILoggerAdapter<EventHubClient> _logger;
private readonly EventHubClientSettings _eventHubClientSettings;
private IMapper _mapper;
private static EventHubProducerClient _equipmentLocClient;
public EventHubClient(ILoggerAdapter<EventHubClient> logger, EventHubClientSettings eventHubClientSettings, IMapper mapper)
{
_logger = logger;
_eventHubClientSettings = eventHubClientSettings;
_mapper = mapper;
_equipmentLocClient = new EventHubProducerClient(_eventHubClientSettings.ConnectionString, _eventHubClientSettings.EquipmentLocation);
}
}
}

Based on my speculation in comments, I'm curious if refactoring to use async/await rather than the explicit continuation in the main loop may help. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while (true)
{
var message = await Task.Factory.StartNew(() => GetText());
var events = new[] { new EventData(Encoding.UTF8.GetBytes(message)) };
await Send(events).ConfigureAwait(false);
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}
If you're set on keeping the continuation, I wonder if a slight structural refactoring in the continuation may help, both to push up creation of the events and to honor the await statements. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while(true)
{
var t = Task.Factory.StartNew(() => GetText());
var _ = t.ContinueWith(async q =>
{
var events = new[] { new EventData(Encoding.UTF8.GetBytes(t.Result)) };
await Send(events).ConfigureAwait(false);
});
await Task.Yield();
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}

What is wrong in this code to throw an ObjectDisposedException?

It's a little hard for me to understand the actual behavior in this scenario. What is actually happening to not execute the task when expected but later when SemaphoreSlim has been disposed? It throws the following exception-
System.ObjectDisposedException {"The semaphore has been disposed."}
I have a class library like -
public class ParallelProcessor
{
private Action[] actions;
private int maxConcurrency;
public ParallelProcessor(Action[] actionList, int maxConcurrency)
{
this.actions = actionList;
this.maxConcurrency = maxConcurrency;
}
public void RunAllActions()
{
if (Utility.IsNullOrEmpty<Action>(actions))
throw new Exception("No Action Found!");
using (SemaphoreSlim concurrencySemaphore = new SemaphoreSlim(maxConcurrency))
{
foreach (Action action in actions)
{
Task.Factory.StartNew(() =>
{
concurrencySemaphore.Wait();
try
{
action();
}
finally
{
concurrencySemaphore.Release();
}
});
}
}
}
}
And I'm using it like-
class Program
{
static void Main(string[] args)
{
int maxConcurrency = 3;
Action[] actions = new Action[] { () => Console.WriteLine(1), () => Console.WriteLine(2), () => Console.WriteLine(3) }; //Array.Empty<Action>();
ParallelProcessor processor = new ParallelProcessor(actions, maxConcurrency);
processor.RunAllActions();
Console.ReadLine();
}
}
Could anybody please shed some light on this? Thanks in advance.

The problem is your using statement. This is how things are happening:
Create the semaphore
Start tasks running in the background
Dispose of the semaphore
Tasks try to use the semaphore... but can't, because it's disposed
Options:
Just remove the using statement (so you don't dispose of the semaphore, but that's unlikely to be a problem unless you're using this really heavily)
Change your method to block (inside the using statement) until all the tasks have completed, e.g. by using Parallel.ForEach instead of calling Task.Factory.StartNew directly
Change your code to dispose of the semaphore in a task which will only execute after all the other tasks have completed

Your semaphore is disposed at the end of the using block, but used by the still running Task created inside it.
I would recommend moving the semaphore up to the class level:
public class ParallelProcessor
{
private Action[] actions;
private SemaphoreSlim concurrencySemaphore;
public ParallelProcessor(Action[] actionList, int maxConcurrency)
{
this.actions = actionList;
concurrencySemaphore = new SemaphoreSlim(maxConcurrency);
}
public void RunAllActions()
{
if (Utility.IsNullOrEmpty<Action>(actions))
throw new Exception("No Action Found!");
foreach (Action action in actions)
{
Task.Factory.StartNew(() =>
{
concurrencySemaphore.Wait();
try
{
action();
}
finally
{
concurrencySemaphore.Release();
}
});
}
}
}
or an alternative approach, where RunAllActions will block until all are done:
public class ParallelProcessor
{
private Action[] actions;
private int maxConcurrency;
public ParallelProcessor(Action[] actionList, int maxConcurrency)
{
this.actions = actionList;
this.maxConcurrency = maxConcurrency;
}
public void RunAllActions()
{
if (Utility.IsNullOrEmpty<Action>(actions))
throw new Exception("No Action Found!");
using (var concurrencySemaphore = new SemaphoreSlim(maxConcurrency))
{
Task.WaitAll(actions.Select(a => Task.Run(() =>
{
concurrencySemaphore.Wait();
try { a(); }
finally { concurrencySemaphore.Release(); }
})).ToArray());
}
}
}

I believe the problem is dispose the concurrencySemaphore which is already in the using statement.
The main use of Using is, it will automatically add try and finally and in finally it will dispose the object that is under using.
https://www.codeproject.com/Articles/6564/Understanding-the-using-statement-in-C
The solution for your case is either remove the using or remove the finally statement

Xamarin.Forms - BeginInvokeOnMainThread for an async Action

I am familiar with the rules about updating UI elements on the UI thread using the Device.BeginInvokeOnMainThread, however I have an operation that needs to be run on the UI thread that is actually a Task.
For example, the Push/PopAsync methods on XLabs.Forms.Mvvm seem to behave incorrectly on iOS unless they are invoked on the UI thread. There is also another example in the Acr.UserDialogs library for displaying toasts etc.
I know that making an Action async is basically creating an async void lambda and runs the risk of creating a deadlock in the case of an exception, obviously I don't want this to happen.
Does anybody have a workaround for performing async operations on the UI thread that doesn't involve marking the Action as async?

Since Xamarin.Forms 4.2 there is now a helper method to run tasks on the main thread and await them.
await Device.InvokeOnMainThreadAsync(SomeAsyncMethod);
Several overloads exist that should cover most scenarios:
System.Threading.Tasks.Task InvokeOnMainThreadAsync (System.Action action);
System.Threading.Tasks.Task InvokeOnMainThreadAsync (System.Func<System.Threading.Tasks.Task> funcTask);
System.Threading.Tasks.Task<T> InvokeOnMainThreadAsync<T> (System.Func<System.Threading.Tasks.Task<T>> funcTask);
System.Threading.Tasks.Task<T> InvokeOnMainThreadAsync<T> (System.Func<T> func);
Related PR: https://github.com/xamarin/Xamarin.Forms/pull/5028
NOTE: the PR had some bug that was fixed in v4.2, so don't use this in v4.1.

Just make sure you handle exceptions in your Action and you should be fine. The problem you described occurs when you don't handle the exceptions. Below is a very simple example of running an async method from the main thread.
private void Test()
{
Device.BeginInvokeOnMainThread(SomeMethod);
}
private async void SomeMethod()
{
try
{
await SomeAsyncMethod();
}
catch (Exception e) // handle whatever exceptions you expect
{
//Handle exceptions
}
}
private async Task SomeAsyncMethod()
{
await Navigation.PushModalAsync(new ContentPage());
}

Dropping this here in case someone wants to await the end of an action which has to be executed on the main thread
public static class DeviceHelper
{
public static Task RunOnMainThreadAsync(Action action)
{
var tcs = new TaskCompletionSource<object>();
Device.BeginInvokeOnMainThread(
() =>
{
try
{
action();
tcs.SetResult(null);
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
public static Task RunOnMainThreadAsync(Task action)
{
var tcs = new TaskCompletionSource<object>();
Device.BeginInvokeOnMainThread(
async () =>
{
try
{
await action;
tcs.SetResult(null);
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
}

Evaluating the Maui framework and the related WeatherTwentyOne demo. The dialogs were throwing exceptions on Android 29 related to auth modifications. Hat tip to this post Why does my app crash after alert dialog buttons are clicked
Anyway, I could not locate the aforementioned async helpers. So I adapted #Dbl's post to include awaitable generic results (or not)
using System;
using System.Threading.Tasks;
using Microsoft.Maui.Controls;
// ReSharper disable AsyncVoidLambda
namespace WeatherTwentyOne.Utils
{
public static class DeviceHelper
{
// https://stackoverflow.com/a/47941859/241296
public static Task<T> RunOnMainThreadAsync<T>(Func<Task<T>> op)
{
var tcs = new TaskCompletionSource<T>();
Device.BeginInvokeOnMainThread(async
() => {
try
{
var t = await op();
tcs.SetResult(t);
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
public static Task RunOnMainThreadAsync(Func<Task> op)
{
var tcs = new TaskCompletionSource();
Device.BeginInvokeOnMainThread(async
() => {
try
{
await op();
tcs.SetResult();
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
}
}
Example usage:
var password = await Utils.DeviceHelper.RunOnMainThreadAsync<string>(async () => await DisplayPromptAsync("Login", "Enter password"));
Edit: 2022-03-26
In response to the following warning:
   BeginInvokeOnMainThread(Action) is obsolete: Use BindableObject.Dispatcher.Dispatch() instead
implementation revised to:
public static class DeviceHelper
{
public static Task<T> RunOnMainThreadAsync<T>(Func<Task<T>> op)
{
var tcs = new TaskCompletionSource<T>();
Application.Current?.Dispatcher.Dispatch(async () =>
{
try
{
var t = await op();
tcs.SetResult(t);
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
public static Task RunOnMainThreadAsync(Func<Task> op)
{
var tcs = new TaskCompletionSource();
Application.Current?.Dispatcher.Dispatch(async () =>
{
try
{
await op();
tcs.SetResult();
}
catch (Exception e)
{
tcs.SetException(e);
}
});
return tcs.Task;
}
}

Instantiate and Call WCF service Asynchronously

I have a helper method I am using to instantiate a WCF service and perform an action. This is working great for synchronous calls and has really cut down on the code in my main class. However, I am trying to implement this same method on an asynchronous call to the service and am having trouble with the syntax.
This is the helper method I am using:
public static void Use(Action<T> action)
{
ChannelFactory<T> Factory = new ChannelFactory<T>("*");
ClientCredentials Credentials = new ClientCredentials();
Credentials.UserName.UserName = USER_NAME;
Credentials.UserName.Password = PASSWORD;
Factory.Endpoint.EndpointBehaviors.Remove(typeof(ClientCredentials));
Factory.Endpoint.EndpointBehaviors.Add(Credentials);
T Client = Factory.CreateChannel();
bool Success = false;
try
{
action(Client);
((IClientChannel)Client).Close();
Factory.Close();
Success = true;
}
catch (CommunicationException cex)
{
Log.Error(cex.Message, cex);
}
catch (TimeoutException tex)
{
Log.Error(tex.Message, tex);
}
finally
{
if (!Success)
{
((IClientChannel)Client).Abort();
Factory.Abort();
}
}
}
This is the synchronous call I am making to the helper method from a timer elapsed event:
async void Timer_Elapsed(object sender, ElapsedEventArgs e)
{
Service<IVehicleService>.Use(client =>
{
Vehicles = client.GetAllVehicles(new GetAllVehiclesRequest()).vehicleList;
});
await UpdateVehicleStatuses();
}
This is where the GetVehicleStatus method is called:
private async Task UpdateVehicleStatuses()
{
// Can the call to GetVehicleStatus be turned into a lambda expression utilizing the helper method?
IEnumerable<Task<VehicleStatus>> StatusQuery = from s in Vehicles
select GetVehicleStatus(s.ClientVehicleId);
List<Task<VehicleStatus>> StatusTasks = StatusQuery.ToList();
...
}
This is the current body of the GetVehicleStatus method:
private async Task<VehicleStatus> GetVehicleStatus(string clientVehicleID)
{
// Can this method be modified to use the helper method?
GetStatusResponse Status = await VehicleClient.GetStatusByClientIdAsync(clientVehicleID);
return Status.vehicleStatus;
}
I would like to apply the same principal from the synchronous call to the asynchronous call so that I don't have to initialize the service in the main class and can encapsulate all of the error handling there. I am having trouble with the syntax when attempting to turn the GetVehicleStatus method into a lambda expression in the UpdateVehicleStatuses method. I have also tried to modify the GetVehicleStatus method to utilize the helper method with no luck. What am I missing?
Thanks!

You'll need an asynchronous version of your helper method:
public static async Task UseAsync(Func<T, Task> action)
{
...
try
{
await action(Client);
...
}
Furthermore, if you need to support return values, then you'll need another overload:
public static async Task<TResult> UseAsync(Func<TClient, Task<TResult>> action)
{
...
TResult result;
try
{
result = await action(Client);
...
return result;
}
Then you can use it as such:
private async Task<VehicleStatus> GetVehicleStatusAsync(string clientVehicleID)
{
GetStatusResponse Status = await UseAsync(client => client.GetStatusByClientIdAsync(clientVehicleID));
return Status.vehicleStatus;
}

Control flow in asynchronous calls

I'm facing difficulties understanding how to handle program control during asynchronous flow.
I have a SessionManager class which calls the initiates the session and we need to register
for the event OnStartApplicationSessionResponse and my control will return to the calling point. I will get the session id in the eventhandler after sometime or the error code if there is an error.
class SessionManager
{
public bool startUp(Object params)
{
try
{
serviceProvider = new ServiceProvider();
serviceProvider.OnStartApplicationSessionResponse += new StartApplicationSessionResponseHandler(ServiceProvider_OnStartApplicationSessionResponse);
serviceProvider.startUp(params);
}
}
public void ServiceProvider_OnStartApplicationSessionResponse(object sender, ServiceProvider.StartApplicationSessionResponseArgs e)
{
//e.getError
//I will get the session Id here or error code
}
}
How do I get sessionId or the error as my control is now at the calling position?

You could use TaskCompletionSource to make the Event awaitable.
class SessionManager
{
private ServiceProvider _serviceProvider;
public int SessionId
{
get;
private set;
}
public Task<bool> StartUp(Object param)
{
_serviceProvider = new ServiceProvider();
var tcs = new TaskCompletionSource<bool>();
_serviceProvider.OnStartApplicationSessionResponse += (sender, args) =>
{
// do your stuff
// e.g.
SessionId = 0xB00B5;
tcs.SetResult(true);
};
_serviceProvider.startUp(param);
return tcs.Task;
}
}
The call would look like:
private static async void SomeButtonClick()
{
var mgr = new SessionManager();
var success = await mgr.StartUp("string");
if (success)
{
Console.WriteLine(mgr.SessionId);
// update ui or whatever
}
}
note: This Feature is available in .Net 4.5.

With the C# feature async and await you are able to rewrite an asynchronous flow into something that is like a synchronous flow. You have only provided some fragments of your code so to provide a complete example I have created some code that resembles your code:
class StartEventArgs : EventArgs {
public StartEventArgs(Int32 sessionId, Int32 errorCode) {
SessionId = sessionId;
ErrorCode = errorCode;
}
public Int32 SessionId { get; private set; }
public Int32 ErrorCode { get; private set; }
}
delegate void StartEventHandler(Object sender, StartEventArgs e);
class ServiceProvider {
public event StartEventHandler Start;
public void Startup(Boolean succeed) {
Thread.Sleep(TimeSpan.FromSeconds(1));
if (succeed)
OnStart(new StartEventArgs(321, 0));
else
OnStart(new StartEventArgs(0, 123));
}
protected void OnStart(StartEventArgs e) {
var handler = Start;
if (handler != null)
handler(this, e);
}
}
The ServiceProvider.Startup method will delay for a second before firing an event that either signals success or failure depending on the succeed parameter provided. The method is rather silly but hopefully is similar to the behavior of your ServiceProvider.Startup method.
You can convert the asynchronous startup into a task using a TaskCompletionSource:
Task<Int32> PerformStartup(ServiceProvider serviceProvider, Boolean succeed) {
var taskCompletionSource = new TaskCompletionSource<Int32>();
serviceProvider.Start += (sender, e) => {
if (e.ErrorCode > 0)
throw new Exception(e.ErrorCode.ToString());
taskCompletionSource.SetResult(e.SessionId);
};
serviceProvider.Startup(succeed);
return taskCompletionSource.Task;
}
Notice how an error signaled by the Start event is converted into an Exception (in production code you should use a custom exception type instead).
Using the async and await feature of C# you can now write code that looks very much like synchronous code even though it actually is asynchronous:
async void Startup(Boolean succeed) {
var serviceProvider = new ServiceProvider();
try {
var sessionId = await PerformStartup(serviceProvider, succeed);
Console.WriteLine(sessionId);
}
catch (Exception ex) {
Console.WriteLine(ex);
}
}
If an error is reported by the Start event you can now deal with in the catch block. Also the session ID is simply a return value of the function. The "magic" is that using await on a Task will return the result of the task when it completes and if an exception is thrown in the task it can be caught on the thread awaiting the task.