TPL .Net Concurrenty Issue with Azure EventHub Producer - c#

I am working with Azure Event Hub producer client and reading messages off of a kafka stream then pass it along to deserialize/map, then pass to Event Hub. I have the consume loop which is creating a task for each consume and then two methods to do processing(this seems to have greatly improved the speed from the kafka lag perspective. However, Event hub makes you create an event batch which I don't necessarily want to use. I just want to send the data one message at a time for now. In order to create a new batch I have to call Dispose(). I am running into an issue where there's another call to the function by the the time I call Dispose() and I get an error saying the object is being used by event hub.
I've also tried using the overload for eventHubProducerClient.SendAsync that allows you to pass in a IEnumerable but i'm running into the same issue with that.
So I believe this to be a synchronization issue, or maybe I need to do a lock somewhere?
Any help would be appreciated.
public void Execute()
{
using (_consumer)
{
try
{
_consumer.Subscribe(_streamConsumerSettings.Topic);
while (true)
{
var result = _consumer.Consume(1000);
if (result == null)
{
continue;
}
var process = Task.Factory.StartNew(() => ProcessMessage(result?.Message?.Value));
var send = process.ContinueWith(t => SendMessage(process.Result));
}
}
catch (ConsumeException e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
_cancelConsume = true;
_consumer.Close();
RestartConsumer();
}
}
}
public static EquipmentJson ProcessMessage(byte[] result)
{
var json = _messageProcessor.DeserializeAndMap(result);
return json;
}
public static void SendMessage(EquipmentJson message)
{
try
{
_eventHubClient.AddToBatch(message);
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
public async Task AddToBatch(EquipmentJson message)
{
if
(!string.IsNullOrEmpty(message.EquipmentLocation))
{
try
{
var batch = await _equipmentLocClient.CreateBatchAsync();
batch.TryAdd(new EventData(Encoding.UTF8.GetBytes(message.EquipmentLocation)));
await _eventHubProducerClient.SendAsync(batch);
batch.Dispose();
_logger.LogInformation($"Data sent {DateTimeOffset.UtcNow}");
}
catch (Exception e)
{
_logger.LogError(e, e.StackTrace ?? e.Message);
}
}
}
public class EventHubClient : IEventHubClient
{
private readonly ILoggerAdapter<EventHubClient> _logger;
private readonly EventHubClientSettings _eventHubClientSettings;
private IMapper _mapper;
private static EventHubProducerClient _equipmentLocClient;
public EventHubClient(ILoggerAdapter<EventHubClient> logger, EventHubClientSettings eventHubClientSettings, IMapper mapper)
{
_logger = logger;
_eventHubClientSettings = eventHubClientSettings;
_mapper = mapper;
_equipmentLocClient = new EventHubProducerClient(_eventHubClientSettings.ConnectionString, _eventHubClientSettings.EquipmentLocation);
}
}
}

Based on my speculation in comments, I'm curious if refactoring to use async/await rather than the explicit continuation in the main loop may help. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while (true)
{
var message = await Task.Factory.StartNew(() => GetText());
var events = new[] { new EventData(Encoding.UTF8.GetBytes(message)) };
await Send(events).ConfigureAwait(false);
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}
If you're set on keeping the continuation, I wonder if a slight structural refactoring in the continuation may help, both to push up creation of the events and to honor the await statements. Perhaps something similar to the following LinqPad snippet:
async Task Main()
{
while(true)
{
var t = Task.Factory.StartNew(() => GetText());
var _ = t.ContinueWith(async q =>
{
var events = new[] { new EventData(Encoding.UTF8.GetBytes(t.Result)) };
await Send(events).ConfigureAwait(false);
});
await Task.Yield();
}
}
public EventHubProducerClient client = new EventHubProducerClient("<< CONNECTION STRING >>");
public async Task Send(EventData[] events)
{
try
{
await client.SendAsync(events).ConfigureAwait(false);
"Sent".Dump();
}
catch (Exception ex)
{
ex.Dump();
}
}
public string GetText()
{
Thread.Sleep(250);
return "Test";
}

Related

How to prepare a console application that pushes and pops into a redis queue?

I need to prepare a console application with 3 buttons, one that adds elements to a Redis queue, one that pops elements out of it and one that displays elements in the queue. I am new to C# and Redis. Can anyone help me with this, or provide me some resources.
I have coded up the connection to the Redis DB and am able to set and get variables.
using System;
using StackExchange.Redis;
namespace RedisConsoleApp1
{
class Program
{
static void Main(string[] args)
{
ConnectionMultiplexer redisCon = ConnectionMultiplexer.Connect("localhost");
IDatabase db = redisCon.GetDatabase();
//db.Lis
db.StringSet("foo", "dog");
string val = db.StringGet("foo");
Console.WriteLine("output is {0}", val);
Console.ReadKey();
}
}
}
Write a channel subscriber like that
public class RedisHostingRunner : HostedService
{
private readonly IServiceProvider _serviceProvider;
IRedisSubscriber _subscriber;
public RedisHostingRunner(IServiceProvider serviceProvider)
{
_serviceProvider = serviceProvider;
_subscriber = _serviceProvider.GetRequiredService<RedisSubscriber>();
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
//while (!cancellationToken.IsCancellationRequested)
//{
_subscriber.SubScribeChannel();
//await Task.Delay(TimeSpan.FromSeconds(60), cancellationToken);
//}
}
public Task ShutdownAsync(CancellationToken cancellationToken = default)
{
return Task.CompletedTask;
}
}
And in your subscriber set a message handler
public void SubScribeChannel()
{
_logger.LogInformation("!SubScribeChannel started!!");
string channelName = _config.ActiveChannelName;
var pubSub = _connectionMultiplexer.GetSubscriber();
try
{
pubSub.Subscribe(channelName, async (channel, message) => await MessageActionAsync(message, channel));
}
catch(Exception ex)
{
_logger.LogInformation(String.Format("!error: {0}", ex.Message));
}
Debug.WriteLine("EOF");
}
In your handler do your job
private async Task MessageActionAsync(RedisValue message, string channel)
{
try
{
Transformer t = new Transformer(_logger);
_logger.LogInformation(String.Format("!SubScribeChannel message received on message!! channel: {0}, message: {1}", channel, message));
string transformedMessage = Transformer.TransformJsonStringData2Message(message);
List<Document> documents = Transformer.Deserialize<List<Document>>(transformedMessage);
await MergeToMongoDb(documents, channel);
_logger.LogInformation("!Merged");
}
catch (Exception ex)
{
_logger.LogInformation(String.Format("!error: {0}", ex.Message));
}
}

Awaitable task in not awaited task

I'm working on a little console application that selects messages from a database queue
and forwards the messages to a rest api (ASP.NET Web Api).
In general the application does the following steps:
Get the number of pending messages
Load the last pending messages
Post the message to the rest api
Remove the message from the database
To make the program more flexible and to have the ability to process every single message
in a separate database transcation the steps 2 - 3 will be executed as tasks.
This means if there are four messages in the database, we'll have four tasks that
should run nearly parallel and process the messages.
This is what the code looks like:
Database message
public class DatabaseMessage
{
public string Message { get; set; }
}
UnitOfWork (Interface IUnitOfWork)
public class UnitOfWork
{
// ... extermely simplified
public int GetNumberOfPendingMessages() { ... }
public DatabaseMessage GetNextPendingMessage() { ... }
public void DeleteMessage(DatabaseMessage message) { ... }
}
HttpService (Interface IHttpService)
public class HttpService
{
private readonly HttpClient _httpClient;
public HttpService()
{
_httpClient = new HttpClient();
/* Some initalization stuff for the HttpClient object */
}
public async Task<HttpResponse> PostMessage(DatabaseMessage message)
{
var content = /* Create content object */
return await _httpClient.PostAsync(..., content);
}
}
MessageProcessingService (Interface IMessageProcessingService)
public class MessageProcessingService
{
private readonly IHttpService _httpService;
private readonly Semaphore _databaseProcessingSemaphore;
public MessageProcessingService(IHttpService httpService)
{
_httpService = httpService;
}
public async Task ProcessDatabaseMessages()
{
var unitOfWork = new UnitOfWork();
var numberOfPendingMessages = unitOfWork.GetNumberOfPendingMessages();
var messageProcessingTasks = new List<Task>();
for(int t = 0; t < numberOfPendingMessages; t++)
{
messageProcessingTasks.Add(new Task(() => {
ProcessMessageAsTask();
}));
}
var continuationHandler = Task.WhenAll(messageProcessingTasks);
messageProcessingTasks.ForEach(e => e.Start());
await continuationHandler;
}
private void ProcessMessageAsTask()
{
// Single unit of work for each tasks
var unitOfWork = new UnitOfWork();
try{
// Starting a database transaction
unitOfWork.StartTransaction();
_databaseProcessingSemaphore.OnWait();
var message = unitOfWork.GetNextPendingMessage();
_databaseProcessingSemaphore.Release();
if(message != null)
{
var response = _httpService.PostMessage(message).Result;
if(response == HttpStatus.OK)
{
unitOfWork.DeleteMessage(message);
unitOfWork.Commit();
}
else
{
unitOfWork.Rollback();
}
}
else
{
unitOfWork.Commit();
}
}
catch(Exception ex)
{
unitOfWork.Rollback();
// Further error handling...
}
}
}
For better understanding, the HttpClient object is created and managed by Unity and is injected
into the MessageProcessingService object. The HttpClient is held as singleton in the container.
I'm facing now the problem that the call of the method _httpService.PostMessage(). For example, if there are five
messages in the message queue, the call fails five times with an exception that tells me that an task has been canceled.
My question is now what is the problem with PostAsync call of the .NET HttpClient? Is the issue caused by the .Result option or would
it be better to create a new instance of the HttpClient for each message processing task?
Or is there a general problem with the architecture with tasks and the processing of rest api calls?
Update 2018-04-04 - 08:09
I've now made the method ProcessMessageAsTask async and I'm awaiting now the call of the HttpService.
But now I don't get any exception at all. In the ressource monitor and while debugging I can see that all tasks reach the call of the HttpClient (return await _httpClient.PostAsync(..., content);)
But there is no exception nor will the messages be posted. But I don't get any exceptions. The program will be closed immediately after the calls of the HttpClient. All futher statements were not processed.
Changes:
public async Task ProcessDatabaseMessages()
{
var unitOfWork = new UnitOfWork();
var numberOfPendingMessages = unitOfWork.GetNumberOfPendingMessages();
var messageProcessingTasks = new List<Task>();
for(int t = 0; t < numberOfPendingMessages; t++)
{
messageProcessingTasks.Add(new Task(async () => {
await ProcessMessageAsTask();
}));
}
var continuationHandler = Task.WhenAll(messageProcessingTasks);
messageProcessingTasks.ForEach(e => e.Start());
await continuationHandler;
}
private async Task ProcessMessageAsTask()
{
// Single unit of work for each tasks
var unitOfWork = new UnitOfWork();
try{
// Starting a database transaction
unitOfWork.StartTransaction();
_databaseProcessingSemaphore.OnWait();
var message = unitOfWork.GetNextPendingMessage();
_databaseProcessingSemaphore.Release();
if(message != null)
{
var response = await _httpService.PostMessage(message);
if(response == HttpStatus.OK)
{
unitOfWork.DeleteMessage(message);
unitOfWork.Commit();
}
else
{
unitOfWork.Rollback();
}
}
else
{
unitOfWork.Commit();
}
}
catch(Exception ex)
{
unitOfWork.Rollback();
// Further error handling...
}
}

Async wait for multiple threads to finish

I have a code block which is eventually accessed by multiple threads. I search for an up to date async mechanism to continue executing when all threads have passed.
Currently I do the following with a CountDownEvent which works just fine (without async support).
public class Watcher
{
private static readonly Logger Log = LogManager.GetCurrentClassLogger();
private readonly CountdownEvent _isUpdating = new CountdownEvent(1);
private readonly IActivity _activity;
public Watcher([NotNull] IActivity activity)
{
_activity = activity ?? throw new ArgumentNullException(nameof(activity));
_activity.Received += OnReceived;
}
private void OnReceived(IReadOnlyCollection<Summary> summaries)
{
_isUpdating.AddCount();
try
{
// Threads processing
}
finally
{
_isUpdating.Signal();
}
}
private void Disable()
{
_activity.Received -= OnReceived;
_isUpdating.Signal();
/* await */ _isUpdating.Wait();
}
}
Do I need to use any of those AsyncCountdownEvent implementations or is there any other built-in mechanism? I already thought about using a BufferBlock because it has async functionality but I think it's a bit overkill.
Additional to the comments:
IActivity is a WebService call (but shouldn't effect the implementation on top or vice versa)
public async Task Start(bool alwayRetry = true, CancellationToken cancellationToken = new CancellationToken())
{
var milliseconds = ReloadSeconds * 1000;
do
{
try
{
var summaries = await PublicAPI.GetSummariesAsync(cancellationToken).ConfigureAwait(false);
OnSummariesReceived(summaries);
}
catch (Exception ex)
{
Log.Error(ex.Message);
OnErrorOccurred(ex);
}
await Task.Delay(milliseconds, cancellationToken).ConfigureAwait(false);
// ReSharper disable once LoopVariableIsNeverChangedInsideLoop
} while (alwayRetry);
}
It's not clear the IActivity signatures; but you can wait for a range of tasks to be completed:
class MultiAsyncTest {
Task SomeAsync1() { return Task.Delay(1000); }
Task SomeAsync2() { return Task.Delay(2000);}
Task EntryPointAsync() {
var tasks = new List<Task>();
tasks.Add(SomeAsync1());
tasks.Add(SomeAsync2());
return Task.WhenAll(tasks);
}
}
What's IActivity's signature? Does it support Task? Or you are using Thread? More explanation would help to a more specified answer.

There is no active ActorContext, this is most likely due to use of async operations from within this actor

I have a problem, and I am not quite sure how to solve this, except for making my Akka Actor not have async methods.
Here is my Actor Code:
public class AggregatorActor : ActorBase, IWithUnboundedStash
{
public IStash Stash { get; set; }
private AggregatorTimer _aggregatorTimer;
private IActorSystemSettings _settings;
private AccountSummary _accountResponse;
private ContactDetails _contactResponse;
private AnalyticDetails _analyticsResponse;
private FinancialDetails _financialResponse;
private ActorSelection _accountActor;
private ActorSelection _contactActor;
private ActorSelection _analyticsActor;
private ActorSelection _financialActor;
public AggregatorActor(IActorSystemSettings settings) : base(settings)
{
_accountActor = Context.System.ActorSelection(ActorPaths.AccountActorPath);
_contactActor = Context.System.ActorSelection(ActorPaths.ContactActorPath);
_analyticsActor = Context.System.ActorSelection(ActorPaths.AnalyticsActorPath);
_financialActor = Context.System.ActorSelection(ActorPaths.FinancialActorPath);
_settings = settings;
}
#region Public Methods
public override void Listening()
{
ReceiveAsync<ProfilerMessages.ProfilerBase>(async x => await HandleMessageAsync(x));
}
private void Busy()
{
Receive<ProfilerMessages.ProfilerBase>(x => Stash.Stash());
}
private void Aggregate()
{
try
{
Context.Sender.Tell(AggregatedSummaryResponse.Instance(_accountResponse, _contactResponse, _analyticsResponse, _financialResponse));
}
catch (Exception ex)
{
ExceptionHandler(ex);
}
}
public override async Task HandleMessageAsync(object msg)
{
//if is summary, generate new isntance of AggregatorTimer in _eventHandlerCollection.
if (msg is ProfilerMessages.GetSummary)
{
//Become busy. Stash
Become(Busy);
//Handle different requests
var clientId = (msg as ProfilerMessages.GetSummary).ClientId;
await HandleSummaryRequest(clientId);
}
}
private async Task HandleSummaryRequest(string clientId)
{
try
{
var accountMsg = new AccountMessages.GetAggregatedData(clientId);
_accountResponse = (await _accountActor.Ask<Messages.AccountMessages.AccountResponseAll>(accountMsg, TimeSpan.FromSeconds(_settings.NumberOfSecondsToWaitForResponse))).AccountDetails;
//Need to uncomment this
var contactMsg = new ContactMessages.GetAggregatedContactDetails(clientId);
_contactResponse = (await _contactActor.Ask<Messages.ContactMessages.ContactResponse>(contactMsg, TimeSpan.FromSeconds(_settings.NumberOfSecondsToWaitForResponse))).ContactDetails;
var analyticMsg = new AnalyticsMessages.GetAggregatedAnalytics(clientId);
_analyticsResponse = (await _analyticsActor.Ask<Messages.AnalyticsMessages.AnalyticsResponse>(analyticMsg, TimeSpan.FromSeconds(_settings.NumberOfSecondsToWaitForResponse))).AnalyticDetails;
var financialMsg = new FinancialMessages.GetAggregatedFinancialDetails(clientId);
_financialResponse = (await _financialActor.Ask<Messages.FinancialMessages.FinancialResponse>(financialMsg, TimeSpan.FromSeconds(_settings.NumberOfSecondsToWaitForResponse))).FinancialDetails;
//Start new timer
_aggregatorTimer = new AggregatorTimer(_settings.NumberOfSecondsToWaitForResponse);
_aggregatorTimer.TimeElapsed += _aggregatorTimer_TimeElapsed;
}
catch (Exception ex)
{
ExceptionHandler(ex);
}
}
//Event that is raised when an external timers time elapsed.
private async void _aggregatorTimer_TimeElapsed(object sender, ElapsedTimeHandlerArg e)
{
Aggregate();
_aggregatorTimer = null;
_accountResponse = null;
_contactResponse = null;
_analyticsResponse = null;
_financialResponse = null;
//Unstash
Stash.Unstash();
//Start listening again
Become(Listening);
}
#endregion
}
Inside the _aggregatorTimer_TimeElapsed event, I call the await Aggregate function, but the following exception is thrown.
There is no active ActorContext, this is most likely due to use of async operations from within this actor
I think this is caused by the fact that the Aggregate function tries to Tell() the Sender about the responses that are aggregated, but those Tasksare not yet completed? I might be completely wrong, but I have no idea why this is thrown.
I'm not sure what do you even need an AggregatorTimer for - in akka you have a Context.System.Scheduler object, which can be used to schedule events going to happen in the future.
Another thing is that you probably shouldn't execute logic inside event handlers. If you really need them in your code, it's better to limit them only to send a message, once an event gets triggered i.e.:
Receive<TimedOut>(_ => /* handle timeout message */);
var self = Self; // bind current self to a variable, so it won't change
_aggregatorTimer.TimeElapsed += (sender, e) => self.Tell(new TimedOut(), self);

Control flow in asynchronous calls

I'm facing difficulties understanding how to handle program control during asynchronous flow.
I have a SessionManager class which calls the initiates the session and we need to register
for the event OnStartApplicationSessionResponse and my control will return to the calling point. I will get the session id in the eventhandler after sometime or the error code if there is an error.
class SessionManager
{
public bool startUp(Object params)
{
try
{
serviceProvider = new ServiceProvider();
serviceProvider.OnStartApplicationSessionResponse += new StartApplicationSessionResponseHandler(ServiceProvider_OnStartApplicationSessionResponse);
serviceProvider.startUp(params);
}
}
public void ServiceProvider_OnStartApplicationSessionResponse(object sender, ServiceProvider.StartApplicationSessionResponseArgs e)
{
//e.getError
//I will get the session Id here or error code
}
}
How do I get sessionId or the error as my control is now at the calling position?
You could use TaskCompletionSource to make the Event awaitable.
class SessionManager
{
private ServiceProvider _serviceProvider;
public int SessionId
{
get;
private set;
}
public Task<bool> StartUp(Object param)
{
_serviceProvider = new ServiceProvider();
var tcs = new TaskCompletionSource<bool>();
_serviceProvider.OnStartApplicationSessionResponse += (sender, args) =>
{
// do your stuff
// e.g.
SessionId = 0xB00B5;
tcs.SetResult(true);
};
_serviceProvider.startUp(param);
return tcs.Task;
}
}
The call would look like:
private static async void SomeButtonClick()
{
var mgr = new SessionManager();
var success = await mgr.StartUp("string");
if (success)
{
Console.WriteLine(mgr.SessionId);
// update ui or whatever
}
}
note: This Feature is available in .Net 4.5.
With the C# feature async and await you are able to rewrite an asynchronous flow into something that is like a synchronous flow. You have only provided some fragments of your code so to provide a complete example I have created some code that resembles your code:
class StartEventArgs : EventArgs {
public StartEventArgs(Int32 sessionId, Int32 errorCode) {
SessionId = sessionId;
ErrorCode = errorCode;
}
public Int32 SessionId { get; private set; }
public Int32 ErrorCode { get; private set; }
}
delegate void StartEventHandler(Object sender, StartEventArgs e);
class ServiceProvider {
public event StartEventHandler Start;
public void Startup(Boolean succeed) {
Thread.Sleep(TimeSpan.FromSeconds(1));
if (succeed)
OnStart(new StartEventArgs(321, 0));
else
OnStart(new StartEventArgs(0, 123));
}
protected void OnStart(StartEventArgs e) {
var handler = Start;
if (handler != null)
handler(this, e);
}
}
The ServiceProvider.Startup method will delay for a second before firing an event that either signals success or failure depending on the succeed parameter provided. The method is rather silly but hopefully is similar to the behavior of your ServiceProvider.Startup method.
You can convert the asynchronous startup into a task using a TaskCompletionSource:
Task<Int32> PerformStartup(ServiceProvider serviceProvider, Boolean succeed) {
var taskCompletionSource = new TaskCompletionSource<Int32>();
serviceProvider.Start += (sender, e) => {
if (e.ErrorCode > 0)
throw new Exception(e.ErrorCode.ToString());
taskCompletionSource.SetResult(e.SessionId);
};
serviceProvider.Startup(succeed);
return taskCompletionSource.Task;
}
Notice how an error signaled by the Start event is converted into an Exception (in production code you should use a custom exception type instead).
Using the async and await feature of C# you can now write code that looks very much like synchronous code even though it actually is asynchronous:
async void Startup(Boolean succeed) {
var serviceProvider = new ServiceProvider();
try {
var sessionId = await PerformStartup(serviceProvider, succeed);
Console.WriteLine(sessionId);
}
catch (Exception ex) {
Console.WriteLine(ex);
}
}
If an error is reported by the Start event you can now deal with in the catch block. Also the session ID is simply a return value of the function. The "magic" is that using await on a Task will return the result of the task when it completes and if an exception is thrown in the task it can be caught on the thread awaiting the task.

Categories