We Keep Coding

TPL .Net Concurrenty Issue with Azure EventHub Producer

I am working with Azure Event Hub producer client and reading messages off of a kafka stream then pass it along to deserialize/map, then pass to Event Hub. I have the consume loop which is creating a task for each consume and then two methods to do processing(this seems to have greatly improved the speed from the kafka lag perspective. However, Event hub makes you create an event batch which I don't necessarily want to use. I just want to send the data one message at a time for now. In order to create a new batch I have to call Dispose(). I am running into an issue where there's another call to the function by the the time I call Dispose() and I get an error saying the object is being used by event hub.
I've also tried using the overload for eventHubProducerClient.SendAsync that allows you to pass in a IEnumerable but i'm running into the same issue with that.
So I believe this to be a synchronization issue, or maybe I need to do a lock somewhere?
Any help would be appreciated.
public void Execute()
{
using (_consumer)
{
try
{
_consumer.Subscribe(_streamConsumerSettings.Topic);
while (true)
{
var result = _consumer.Consume(1000);
if (result == null)
{
continue;
}
var process = Task.Factory.StartNew(() => ProcessMessage(result?.Message?.Value));
var send = process.ContinueWith(t => SendMessage(process.Result));
}
}
catch (ConsumeException e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
_cancelConsume = true;
_consumer.Close();
RestartConsumer();
}
}
}
public static EquipmentJson ProcessMessage(byte[] result)
{
var json = _messageProcessor.DeserializeAndMap(result);
return json;
}
public static void SendMessage(EquipmentJson message)
{
try
{
_eventHubClient.AddToBatch(message);
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
public async Task AddToBatch(EquipmentJson message)
{
if
(!string.IsNullOrEmpty(message.EquipmentLocation))
{
try
{
var batch = await _equipmentLocClient.CreateBatchAsync();
batch.TryAdd(new EventData(Encoding.UTF8.GetBytes(message.EquipmentLocation)));
await _eventHubProducerClient.SendAsync(batch);
batch.Dispose();
_logger.LogInformation($"Data sent {DateTimeOffset.UtcNow}");
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
}
public class EventHubClient : IEventHubClient
{
private readonly ILoggerAdapter<EventHubClient> _logger;
private readonly EventHubClientSettings _eventHubClientSettings;
private IMapper _mapper;
private static EventHubProducerClient _equipmentLocClient;
public EventHubClient(ILoggerAdapter<EventHubClient> logger, EventHubClientSettings eventHubClientSettings, IMapper mapper)
{
_logger = logger;
_eventHubClientSettings = eventHubClientSettings;
_mapper = mapper;
_equipmentLocClient = new EventHubProducerClient(_eventHubClientSettings.ConnectionString, _eventHubClientSettings.EquipmentLocation);
}
}
}

Based on my speculation in comments, I'm curious if refactoring to use async/await rather than the explicit continuation in the main loop may help. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while (true)
{
var message = await Task.Factory.StartNew(() => GetText());
var events = new[] { new EventData(Encoding.UTF8.GetBytes(message)) };
await Send(events).ConfigureAwait(false);
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}
If you're set on keeping the continuation, I wonder if a slight structural refactoring in the continuation may help, both to push up creation of the events and to honor the await statements. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while(true)
{
var t = Task.Factory.StartNew(() => GetText());
var _ = t.ContinueWith(async q =>
{
var events = new[] { new EventData(Encoding.UTF8.GetBytes(t.Result)) };
await Send(events).ConfigureAwait(false);
});
await Task.Yield();
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}

How to run backgroundservices on seperate thread?

I realized that background services blocks main thread and suddenly it makes latency in my main thread.I can't find useful resources about how to make backgroundservices to not make latency and block main thread.Any suggestions?
public Task StartAsync(CancellationToken cancellationToken)
{
_timer = new Timer(DoWork, null, 0, 10000);
Initialize().GetAwaiter();
return Task.CompletedTask;
}
private async void DoWork(object state)
{
try
{
//some logic
await testService.DoWorkAsync();
}
catch (Exception ex)
{
Log.Error(ex, "TestBackgroundService DoWork throws ");
}
}
private async Task Initialize()
{
while (true)
{
try
{
if (IsEnabled)
{
//some logic
await testService.PrintAsync();
IsEnabled = false;
}
else
{
//some logic
await testService.CloseAsync();
if (condition)
{
IsEnabled = true;
}
}
await Task.Delay(1000);
}
catch (Exception ex)
{
Log.Error($"ExecuteAsync throws {ex.ToString()}");
}
}
}

You have to use it the right way and I think scoped background task is what you want:
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio#consuming-a-scoped-service-in-a-background-task
Here you have a ExecuteAsync Method and you could use it like this:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
//your code
await DoWork(stoppingToken);
Task.Delay(1000);
}
}

If it is possible, you can create an Controller for the background jobs.
From your code, you can simply invoke the controller action using an HttpClient method.
private async Task ExecuteJobAsync(string jobName)
{
// background job
string uri = $"http://localhost:21454/api/jobs/{jobName}";
using (HttpClient client = new HttpClient())
{
using (HttpContent content = new StringContent(string.Empty, Encoding.UTF8, "application/json"))
{
using (HttpResponseMessage response = await httpClient.PostAsync(uri, content).ConfigureAwait(false))
{
}
}
}
}
Invoking background job:
private async Task Initialize()
{
while (true)
{
try
{
if (IsEnabled)
{
await ExecuteJobAsync("print");
IsEnabled=false;
}
else
{
//some logic
await ExecuteJobAsync("close");
if(condition){
IsEnabled=true;
}
}
await Task.Delay(1000);
}
catch (Exception ex)
{
Log.Error($"ExecuteAsync throws {ex.ToString()}");
}
}
}
Jobs controller:
[Route("api/jobs")]
public class JobsController : Controller
{
[HttpPost("print")]
public void PrintAction()
{
// code
}
[HttpPost("close")]
public void CloseAction()
{
// code
}
}

When I call an Async method what return type should I use to indicate success?

Here's what I have. There's more to the code but I just put in this snippet to make the question smaller.
public async Task GetCards(CancellationTokenSource cts)
{
try
{
while (true)
{
await CheckCardAvailability(cts);
}
}
catch (TaskCanceledException)
{
cts.Dispose();
}
}
private async Task CheckCardAvailability(CancellationTokenSource cts)
{
if (Counts.selectedCardCount == 0)
{
CardCountZeroMessages();
SetMessageView(false);
await Task.Delay(500, cts.Token);
return;
}
if (XYX == 0)
{
// other code
await Task.Delay(500, cts.Token);
return;
}
}
I just have a simple return here but should I do this differently as the return type is Task? Note that I added the return as I have many of these if conditions and I didn't want to do a lot of if else if else if ..

As mentioned by #PanagiotisKanavos, you might consider changing your code to:
public async Task GetCards(CancellationTokenSource cts)
{
try
{
while (true)
{
CheckCardAvailability(cts);
await Task.Delay(500, cts.Token); // or something with a timer
}
}
catch (TaskCanceledException)
{
cts.Dispose();
}
}
private /*async Task*/void CheckCardAvailability(CancellationTokenSource cts)
{
if (Counts.selectedCardCount == 0)
{
CardCountZeroMessages();
SetMessageView(false);
// await Task.Delay moved outside
}
if (XYX == 0)
{
// other code
// await Task.Delay moved outside
}
}

Remove cancelled Task from producer/consumer queue

I want to use an async producer/consumer queue (AsyncEx lib) to send messages one at a time over a bus. Right now I achieve this simply by async blocking. It's working fine, but I have no control over the queue :(
So I came up with following solution, problem is that a canceled task is not removed from the queue. If I limit the queue to say 10 (because each message takes 1s to send and max queue time shall be 10s or so) and the queue contains already 8 waiting tasks and 2 canceled tasks, than the next queued task would throw an InvalidOperationException although the two canceled task wouldn't be sent anyway.
Maybe there is a better way to do this :D
class Program
{
static AsyncProducerConsumerQueue<Tuple<string, TaskCompletionSource>> s_Queue =
new AsyncProducerConsumerQueue<Tuple<string, TaskCompletionSource>>();
static void Main()
{
StartAsync().Wait();
}
static async Task StartAsync()
{
var sendingTask = StartSendingAsync();
var tasks = new List<Task>();
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(8)))
{
for (var i = 0; i < 10; i++)
{
tasks.Add(EnqueueMessageAsync("Message " + i, cts.Token));
}
try
{
await Task.WhenAll(tasks);
Console.WriteLine("All messages sent.");
}
catch (TaskCanceledException)
{
Console.WriteLine("At least one task was canceled.");
}
}
s_Queue.CompleteAdding();
await sendingTask;
s_Queue.Dispose();
Console.WriteLine("Queue completed.");
Console.WriteLine("Press any key to continue...");
Console.ReadKey();
}
static async Task EnqueueMessageAsync(string message, CancellationToken token)
{
var tcs = new TaskCompletionSource();
using (token.Register(() => tcs.TrySetCanceled()))
{
await s_Queue.EnqueueAsync(new Tuple<string, TaskCompletionSource>(message, tcs));
Console.WriteLine("Thread '{0}' - {1}: {2} queued.", Thread.CurrentThread.ManagedThreadId, DateTime.Now.TimeOfDay, message);
await tcs.Task;
}
}
static async Task SendMessageAsync(string message)
{
await Task.Delay(TimeSpan.FromSeconds(1));
Console.WriteLine("Thread '{0}' - {1}: {2} sent.", Thread.CurrentThread.ManagedThreadId, DateTime.Now.TimeOfDay, message);
}
static async Task StartSendingAsync()
{
while (await s_Queue.OutputAvailableAsync())
{
var t = await s_Queue.DequeueAsync();
if (t.Item2.Task.IsCanceled || t.Item2.Task.IsFaulted) continue;
await SendMessageAsync(t.Item1);
t.Item2.TrySetResult();
}
}
}
Edit 1:
As svik pointed out the InvalidOperationException is only thrown if the queue is already completed. So this solution doesn't even solve my initial problem of an unmanaged "queue" of waiting tasks. If there are e.g. more than 10 calls/10s I got a full queue and an additional unmanaged "queue" of waiting tasks like with my async blocking approach (AsyncMonitor). I guess I have to come up with some other solution then...
Edit 2:
I have N different producers of messages (I don't know how many there are because it's not my code) and only one consumer that sends the messages over a bus and checks if they were sent correctly (not really string messages).
The following code simulates a situation where the code should break (queue size is 10):
Enqueue 10 messages (with an timeout of 5sec)
Wait 5sec (message 0-4 were sent and message 5-9 were cancelled)
Enqueue 11 new messages (w/o timeout)
Message 10 - 19 should be enqueued because the queue only contains cancelled messages
Message 20 should throw an exception (e.g. QueueOverflowException) because the queue is full, this would be handled or not by the producer code
Producers:
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
{
for (var i = 0; i < 10; i++) { tasks.Add(EnqueueMessageAsync("Message " + i, cts.Token)); }
await Task.Delay(TimeSpan.FromSeconds(5));
for (var i = 10; i < 21; i++) { tasks.Add(EnqueueMessageAsync("Message " + i, default(CancellationToken))); }
try
{
await Task.WhenAll(tasks);
Console.WriteLine("All messages sent.");
}
catch (TaskCanceledException)
{
Console.WriteLine("At least one task was canceled.");
Console.WriteLine("Press any key to complete queue...");
Console.ReadKey();
}
}
The goal is, I want to have full control over all messages that should be send, but this is not the case in the code I've posted before, because I only have control over the messages in the queue but not the messages that are waiting to be enqueued (there could be 10000 messages asynchronously waiting to be enqueued and I wouldn't know => producer code wouldn't work as expected anyway because it would take forever to send all the messages that are waiting...)
I hope this makes it clearer what I want to achieve ;)

I'm not sure if answering my own questing is OK, so I won't flag it as answer, maybe someone comes up with a better solution :P
First of all here is the producer code:
static async Task StartAsync()
{
using (var queue = new SendMessageQueue(10, new SendMessageService()))
using (var timeoutTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(4.5)))
{
var tasks = new List<Task>();
for (var i = 0; i < 10; i++)
{
tasks.Add(queue.SendAsync(i.ToString(), timeoutTokenSource.Token));
}
await Task.Delay(TimeSpan.FromSeconds(4.5));
for (var i = 10; i < 25; i++)
{
tasks.Add(queue.SendAsync(i.ToString(), default(CancellationToken)));
}
await queue.CompleteSendingAsync();
for (var i = 0; i < tasks.Count; i++ )
{
try
{
await tasks[i];
Console.WriteLine("Message '{0}' send.", i);
}
catch (TaskCanceledException)
{
Console.WriteLine("Message '{0}' canceled.", i);
}
catch (QueueOverflowException ex)
{
Console.WriteLine(ex.Message);
}
}
}
Console.WriteLine("Press any key to continue...");
Console.ReadKey();
}
25 messages are enqueued over 5sec
16 messages are sent
3 messages are not sent (queue is full)
6 messages get canceled
And here is the "Queue" class that is based upon a List. It's a combination of the queue and the consumer. Synchronisation is done with the AsyncMonitor class (AsyncEx by Stephen Cleary).
class SendMessageQueue : IDisposable
{
private bool m_Disposed;
private bool m_CompleteSending;
private Task m_SendingTask;
private AsyncMonitor m_Monitor;
private List<MessageTaskCompletionSource> m_MessageCollection;
private ISendMessageService m_SendMessageService;
public int Capacity { get; private set; }
public SendMessageQueue(int capacity, ISendMessageService service)
{
Capacity = capacity;
m_Monitor = new AsyncMonitor();
m_MessageCollection = new List<MessageTaskCompletionSource>();
m_SendMessageService = service;
m_SendingTask = StartSendingAsync();
}
public async Task<bool> SendAsync(string message, CancellationToken token)
{
if (m_Disposed) { throw new ObjectDisposedException(GetType().Name); }
if (message == null) { throw new ArgumentNullException("message"); }
using (var messageTcs = new MessageTaskCompletionSource(message, token))
{
await AddAsync(messageTcs);
return await messageTcs.Task;
}
}
public async Task CompleteSendingAsync()
{
if (m_Disposed) { throw new ObjectDisposedException(GetType().Name); }
using (m_Monitor.Enter())
{
m_CompleteSending = true;
}
await m_SendingTask;
}
private async Task AddAsync(MessageTaskCompletionSource message)
{
using (await m_Monitor.EnterAsync(message.Token))
{
if (m_CompleteSending) { throw new InvalidOperationException("Queue already completed."); }
if (Capacity < m_MessageCollection.Count)
{
m_MessageCollection.RemoveAll(item => item.IsCanceled);
if (Capacity < m_MessageCollection.Count)
{
throw new QueueOverflowException(string.Format("Queue overflow; '{0}' couldn't be enqueued.", message.Message));
}
}
m_MessageCollection.Add(message);
}
m_Monitor.Pulse(); // signal new message
Console.WriteLine("Thread '{0}' - {1}: '{2}' enqueued.", Thread.CurrentThread.ManagedThreadId, DateTime.Now.TimeOfDay, message.Message);
}
private async Task<MessageTaskCompletionSource> TakeAsync()
{
using (await m_Monitor.EnterAsync())
{
var message = m_MessageCollection.ElementAt(0);
m_MessageCollection.RemoveAt(0);
return message;
}
}
private async Task<bool> OutputAvailableAsync()
{
using (await m_Monitor.EnterAsync())
{
if (m_MessageCollection.Count > 0) { return true; }
else if (m_CompleteSending) { return false; }
await m_Monitor.WaitAsync();
return true;
}
}
private async Task StartSendingAsync()
{
while (await OutputAvailableAsync())
{
var message = await TakeAsync();
if (message.IsCanceled) continue;
try
{
var result = await m_SendMessageService.SendMessageAsync(message.Message, message.Token);
message.TrySetResult(result);
}
catch (TaskCanceledException) { message.TrySetCanceled(); }
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected void Dispose(bool disposing)
{
if (m_Disposed) return;
if (disposing)
{
if (m_MessageCollection != null)
{
var tmp = m_MessageCollection;
m_MessageCollection = null;
tmp.ForEach(item => item.Dispose());
tmp.Clear();
}
}
m_Disposed = true;
}
#region MessageTaskCompletionSource Class
class MessageTaskCompletionSource : TaskCompletionSource<bool>, IDisposable
{
private bool m_Disposed;
private IDisposable m_CancellationTokenRegistration;
public string Message { get; private set; }
public CancellationToken Token { get; private set; }
public bool IsCompleted { get { return Task.IsCompleted; } }
public bool IsCanceled { get { return Task.IsCanceled; } }
public bool IsFaulted { get { return Task.IsFaulted; } }
public MessageTaskCompletionSource(string message, CancellationToken token)
{
m_CancellationTokenRegistration = token.Register(() => TrySetCanceled());
Message = message;
Token = token;
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected void Dispose(bool disposing)
{
if (m_Disposed) return;
if (disposing)
{
TrySetException(new ObjectDisposedException(GetType().Name));
if (m_CancellationTokenRegistration != null)
{
var tmp = m_CancellationTokenRegistration;
m_CancellationTokenRegistration = null;
tmp.Dispose();
}
}
m_Disposed = true;
}
}
#endregion
}
For now I'am OK with this solution; it's gets the job done :D

Task Parallel library waiting for method in loop to finish (.NET 4.0)

wonder if you can help.
I have a listview displaying a list of Customer EG (Customer1,Customer2,Customer3 etc..)
Now each customer can have 1000s of orders to process so
The problem I have is that I would like to wait for processcustomer to finish before starting with new one,and
I dont know how to code it to make my "ProcessCustomer" in the loop to finish before can proceed with the next one.
Psuedo code below:
public class form1
{
private void StartProcessingCustomer_Click(object sender, EventArgs e)
{
foreach (ListViewItem item in lvwCustomers.Items)
{
customerPresenter.ProcessCustomer(item.Tag as Customer);
}
}
}
public class CustomerPresenter
{
private void ProcessCustomer(Customer customer)
{
CustomerProcessor customerProcessor=new CustomerProcessor();
customerProcessor.OrderProcessing += OnCustomerProcessing;
customerProcessor.OrderProcessed += OnCustomerProcessed;
customerProcessor.TaskCompleted += OnCustomerCompleted;
customerProcessor.ProcessCustomer(customer);
}
}
public class CustomerProcessor
{
public CustomerProcessor()
{
}
public void ProcessCustomer(Customer customer)
{
//simplified version with psuedo code
cancelToken = new CancellationTokenSource();
var parOpts = new ParallelOptions();
parOpts.CancellationToken = cancelToken.Token;
parOpts.MaxDegreeOfParallelism = -1;
const int index = 0;
try
{
sw.Start();
processResult.StartedAt = DateTime.Now;
Task t1 = Task.Factory.StartNew(() =>
{
try
{
Parallel.ForEach(customer.Orders, parOpts, (order, loopState) =>
{
count = Interlocked.Increment(ref count);
ProcessOrder(order, count);
});
}
catch (OperationCanceledException)
{
//omitted code
}
}, cancelToken.Token).ContinueWith(result => OnTaskCompleted(
{
//omitted code
}), new CancellationTokenSource().Token, TaskContinuationOptions.None, TaskScheduler.FromCurrentSynchronizationContext());
}
catch (AggregateException ae)
{
//omitted code
}
}
private void ProcessOrder(Order order, int index)
{
//we process the order here
}
}
}

If you make your ProcessCustomer method return a Task, you can asynchronously wait for it's completion:
public Task ProcessCustomer(Customer customer)
{
return Task.Factory.StartNew(() =>
{
try
{
Parallel.ForEach(customer.Orders, parOpts, (order, loopState) =>
{
count = Interlocked.Increment(ref count);
ProcessOrder(order, count);
});
}
catch (OperationCanceledException)
{
//omitted code
}
}, cancelToken.Token).ContinueWith(result => OnTaskCompleted(
{
//omitted code
}), new CancellationTokenSource().Token, TaskContinuationOptions.None,
TaskScheduler.FromCurrentSynchronizationContext());
}
Now you can:
//I'm assuming this is an event handler.
private async void SomeEventHandler(object sender, EventArgs e)
{
foreach (ListViewItem item in lvwCustomers.Items)
{
await customerPresenter.ProcessCustomer(item.Tag as Customer);
}
}
As you're using .NET 4.0, you'll need to use Microsoft.Bcl.Async to make async-await available.