How to re-enqueue a failed background async task - c#

Good day, I'm trying to create a general purpose task queue service that executes in the background using BackgroundService. I avoided using a Delegate function Func<T1,T2, OuT> as input to EnqueueTask(Task<ResponseHelper> newTask) method because I wanted a generic solution, so I opted to pass a Task<ResponseHelper> instead. But this solution does not allow me to re-enqueue a failed task inside ExecuteAsync(CancellationToken stoppingToken) because the task returns an instance of ResponseHelper which is not a copy of the failed task. Kindly help me correct the code I have to return a dequeued task from DequeueTaskAsync(CancellationToken cancellationToken) function instead of an instance of ResponseHelper.
public interface ITaskQueueHelper
{
void EnqueueTask(Task<ResponseHelper> newTask);
Task<ResponseHelper> DequeueTaskAsync(CancellationToken cancellationToken);
}
public class TaskQueueHelper : ITaskQueueHelper
{
private readonly SemaphoreSlim signal;
private readonly ConcurrentQueue<Task<ResponseHelper>> taskQueue;
public TaskQueueHelper()
{
signal = new SemaphoreSlim(0);
taskQueue = new ConcurrentQueue<Task<ResponseHelper>>();
}
public void EnqueueTask(Task<ResponseHelper> newTask)
{
if (newTask == null)
{
throw new ArgumentNullException(nameof(newTask));
}
taskQueue.Enqueue(newTask);
signal.Release();
}
public async Task<ResponseHelper> DequeueTaskAsync(CancellationToken cancellationToken)
{
await signal.WaitAsync(cancellationToken);
taskQueue.TryDequeue(out var currentTask);
/*I need to return currentTask here, instead of an instance of ResponseHelper*/
return await currentTask;
}
}
public class TaskQueueService : BackgroundService
{
private readonly ITaskQueueHelper taskQueue;
private readonly ILogger<TaskQueueService> logger;
public TaskQueueService(
ITaskQueueHelper _taskQueue,
ILogger<TaskQueueService> _logger)
{
logger = _logger;
taskQueue = _taskQueue;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
ResponseHelper response = await taskQueue.DequeueTaskAsync(stoppingToken);
try
{
if (!response.Status.Equals(ResultCode.Success))
{
/*I need to re-enqueue a failed task here*/
//taskQueue.EnqueueTask(currentTask);
}
}
catch (Exception e)
{
logger.LogError(e, $"Error occurred executing {nameof(TaskQueueService)}");
}
}
}
}

for retrying you could do:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
ResponseHelper response = await taskQueue.DequeueTaskAsync(stoppingToken);
try
{
if (!response.Status.Equals(ResultCode.Success))
{
// Retry the task.
response = await taskQueue.DequeueTaskAsync(stoppingToken);
}
}
catch (Exception e)
{
logger.LogError(e, $"Error occurred executing {nameof(TaskQueueService)}");
}
}
}

Related

Task queue without waiting for results

I am wondering if anyone can help me I am trying to write a class which will create a task queue of tasks which runs without waiting for the result. I have a couple issues the first is I want to remove the completed tasks from the queue so that the task queue does not grow to large and when application shuts down it will clear/finish the task queue first.
public interface IBackgroundTaskQueue
{
void QueueBackgroundWorkItem(Task? workItem, CancellationToken cancellationToken);
Task<Task?> DequeueAsync(
CancellationToken cancellationToken);
}
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private readonly ConcurrentQueue<Task?> _workItems = new();
private readonly ILogger<BackgroundTaskQueue> _logger;
private readonly SemaphoreSlim _signal = new(0);
public BackgroundTaskQueue(ILogger<BackgroundTaskQueue> logger, IHostApplicationLifetime lifetime)
{
_logger = logger;
lifetime.ApplicationStopped.Register(OnStopping);
lifetime.ApplicationStopping.Register(OnStopping);
}
public void QueueBackgroundWorkItem(Task? workItem, CancellationToken cancellationToken)
{
if (workItem is null or { IsCanceled: true })
return;
_workItems.Enqueue(workItem);
_signal.Release();
}
private void OnStopping()
{
try
{
DequeueAsync(CancellationToken.None).GetAwaiter().GetResult();
}
catch (Exception ex)
{
_logger.LogWarning(ex, "An error has occurred whilst attempting to dequeue the background work que.");
}
}
public async Task<Task?> DequeueAsync(CancellationToken cancellationToken)
{
try
{
await _signal.WaitAsync(cancellationToken);
_workItems.TryDequeue(out var workItem);
return workItem;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "An error has occurred whilst attempting to dequeue the background work que.");
}
return null;
}
}

Inheriting from Stephen Cleary's Nito.Disposables NuGet package

I saw the implementation of Stephen Cleary's Disposables NuGet package and it seems like it's perfect in my case, even tho, I couldn't find examples on how to inherit from it.
My idea is to make UnsubscribeAsync().GetAwaiter().GetResult(); to await UnsubscribeAsync();, which means it should be wrapped into an IAsyncDisposable. How could I achieve that with a sealed class?
public sealed class LiveTradeManager : ITradeManager, IDisposable
{
private bool _disposed;
private readonly ILogger<LiveTradeManager> _logger;
private readonly TradeOptions _tradeOptions;
private readonly IBotClient _client;
private string _listenKey;
private UpdateSubscription _candleSubscription, _accountUpdateSubscription;
private IDictionary<string, Channel<IBinanceStreamKlineData>> _channels;
public LiveTradeManager(ILogger<LiveTradeManager> logger, IOptions<TradeOptions> tradeOptions, IOptions<ExchangeOptions> exchangeOptions, IBotClientFactory clientFactory)
{
_logger = logger;
_tradeOptions = tradeOptions.Value;
_client = clientFactory.GetBotClient(exchangeOptions.Value.BotClientType);
}
public bool IsPaused { get; set; }
public async Task RunAsync(CancellationToken cancellationToken)
{
try
{
await SubscribeAsync(cancellationToken).ConfigureAwait(false);
}
catch (TaskCanceledException)
{
}
catch (Exception ex) when (Handle(() => _logger.LogError(ex, "Unexpected error.")))
{
}
}
private async Task SubscribeAsync(CancellationToken cancellationToken)
{
// Subscribe to account updates
_listenKey = await _client.GetListenKeyAsync(cancellationToken).ConfigureAwait(false);
void OnOrderUpdate(BinanceStreamOrderUpdate order)
{
// order update logic
}
_accountUpdateSubscription = await _client.SubscribeToUserDataUpdatesAsync(_listenKey, OnOrderUpdate).ConfigureAwait(false);
_ = Task.Run(async () =>
{
while (!cancellationToken.IsCancellationRequested)
{
await _client.KeepAliveListenKeyAsync(_listenKey, cancellationToken).ConfigureAwait(false);
await Task.Delay(TimeSpan.FromMinutes(30), cancellationToken).ConfigureAwait(false);
}
}, cancellationToken);
// Subscribe to candle updates
var symbols = _tradeOptions.Symbols.Select(x => x.ToString()).ToList();
_channels = symbols.ToDictionary(x => x, _ =>
Channel.CreateBounded<IBinanceStreamKlineData>(new BoundedChannelOptions(1)
{FullMode = BoundedChannelFullMode.DropOldest}));
async void OnCandleReceived(IBinanceStreamKlineData data)
{
if (IsPaused) return;
try
{
var ohlcv = data.Data.ToCandle();
if (data.Data.Final)
{
_logger.LogInformation(
$"[{data.Symbol}] Finalized candle | Open time: {ohlcv.Timestamp.ToDateTimeFormat()} | Price: {ohlcv.Close}");
_ = Task.Run(async () =>
{
await Task.Delay(10000, cancellationToken).ConfigureAwait(false);
}, cancellationToken);
}
else
{
_logger.LogInformation(
$"[{data.Symbol}] Candle update | Open time: {ohlcv.Timestamp.ToDateTimeFormat()} | Price: {ohlcv.Close}");
await _channels[data.Symbol].Writer.WriteAsync(data, cancellationToken).ConfigureAwait(false);
}
}
catch (TaskCanceledException)
{
}
}
_candleSubscription = await _client
.SubscribeToCandleUpdatesAsync(symbols, KlineInterval.OneMinute, OnCandleReceived)
.ConfigureAwait(false);
var tasks = _channels.Values.Select(async channel =>
{
await foreach (var data in channel.Reader.ReadAllAsync(cancellationToken))
{
// long-running logic...
await Task.Delay(10000, cancellationToken).ConfigureAwait(false);
}
});
// NOTE: this would block further logic
await Task.WhenAll(tasks).ConfigureAwait(false);
}
private async Task UnsubscribeAsync()
{
// Unsubscribe account updates
if (!string.IsNullOrEmpty(_listenKey))
{
await _client.StopListenKeyAsync(_listenKey).ConfigureAwait(false);
}
if (_accountUpdateSubscription != null)
{
await _client.UnsubscribeAsync(_accountUpdateSubscription).ConfigureAwait(false);
}
// Unsubscribe candle updates
if (_candleSubscription != null)
{
await _client.UnsubscribeAsync(_candleSubscription).ConfigureAwait(false);
}
// Channels
if (_channels != null)
{
foreach (var channel in _channels.Values)
{
channel.Writer.Complete();
}
_channels.Clear();
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
if (_disposed) return;
if (disposing)
{
UnsubscribeAsync().GetAwaiter().GetResult();
}
_disposed = true;
}
}
public class BotManagerService : BackgroundService
{
private readonly IHostApplicationLifetime _hostApplicationLifetime;
private readonly IDiscordClient _discordClient;
private readonly ITradeManager _tradeManager;
public BotManagerService(
IHostApplicationLifetime hostApplicationLifetime,
IOptions<ExchangeOptions> options,
IDiscordClient discordClient,
ITradeManagerFactory tradeManagerFactory)
{
_hostApplicationLifetime = hostApplicationLifetime;
_discordClient = discordClient;
_tradeManager = tradeManagerFactory.GetTradeManager(options.Value.TradeManagerType);
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
var task1 = _tradeManager.RunAsync(stoppingToken);
var task2 = _discordClient.StartAsync();
await Task.WhenAll(task1, task2).ConfigureAwait(false);
}
finally
{
_hostApplicationLifetime.StopApplication();
}
}
}
I couldn't find examples on how to inherit from it.
Nito.Disposables, like the vast majority of the code I write, is written for composition rather than inheritance.
So, if you have a type that needs to implement IAsyncDisposable, it should contain an IAsyncDisposable implementation and forward its interface methods to that contained object:
public sealed class LiveTradeManager : ITradeManager, IAsyncDisposable
{
private readonly AsyncDisposable _disposable;
...
public LiveTradeManager(...)
{
...
_disposable = new(async () => await UnsubscribeAsync());
}
public ValueTask DisposeAsync() => _disposable.DisposeAsync();
}

How to run backgroundservices on seperate thread?

I realized that background services blocks main thread and suddenly it makes latency in my main thread.I can't find useful resources about how to make backgroundservices to not make latency and block main thread.Any suggestions?
public Task StartAsync(CancellationToken cancellationToken)
{
_timer = new Timer(DoWork, null, 0, 10000);
Initialize().GetAwaiter();
return Task.CompletedTask;
}
private async void DoWork(object state)
{
try
{
//some logic
await testService.DoWorkAsync();
}
catch (Exception ex)
{
Log.Error(ex, "TestBackgroundService DoWork throws ");
}
}
private async Task Initialize()
{
while (true)
{
try
{
if (IsEnabled)
{
//some logic
await testService.PrintAsync();
IsEnabled = false;
}
else
{
//some logic
await testService.CloseAsync();
if (condition)
{
IsEnabled = true;
}
}
await Task.Delay(1000);
}
catch (Exception ex)
{
Log.Error($"ExecuteAsync throws {ex.ToString()}");
}
}
}
You have to use it the right way and I think scoped background task is what you want:
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio#consuming-a-scoped-service-in-a-background-task
Here you have a ExecuteAsync Method and you could use it like this:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
//your code
await DoWork(stoppingToken);
Task.Delay(1000);
}
}
If it is possible, you can create an Controller for the background jobs.
From your code, you can simply invoke the controller action using an HttpClient method.
private async Task ExecuteJobAsync(string jobName)
{
// background job
string uri = $"http://localhost:21454/api/jobs/{jobName}";
using (HttpClient client = new HttpClient())
{
using (HttpContent content = new StringContent(string.Empty, Encoding.UTF8, "application/json"))
{
using (HttpResponseMessage response = await httpClient.PostAsync(uri, content).ConfigureAwait(false))
{
}
}
}
}
Invoking background job:
private async Task Initialize()
{
while (true)
{
try
{
if (IsEnabled)
{
await ExecuteJobAsync("print");
IsEnabled=false;
}
else
{
//some logic
await ExecuteJobAsync("close");
if(condition){
IsEnabled=true;
}
}
await Task.Delay(1000);
}
catch (Exception ex)
{
Log.Error($"ExecuteAsync throws {ex.ToString()}");
}
}
}
Jobs controller:
[Route("api/jobs")]
public class JobsController : Controller
{
[HttpPost("print")]
public void PrintAction()
{
// code
}
[HttpPost("close")]
public void CloseAction()
{
// code
}
}

Background task of writing to the database by timer

How to write to the database on a timer in the background. For example, check mail and add new letters to the database. In the example, I simplified the code just before writing to the database.
The class names from the example in Microsoft.
The recording class itself:
namespace EmailNews.Services
{
internal interface IScopedProcessingService
{
void DoWork();
}
internal class ScopedProcessingService : IScopedProcessingService
{
private readonly ApplicationDbContext _context;
public ScopedProcessingService(ApplicationDbContext context)
{
_context = context;
}
public void DoWork()
{
Mail mail = new Mail();
mail.Date = DateTime.Now;
mail.Note = "lala";
mail.Tema = "lala";
mail.Email = "lala";
_context.Add(mail);
_context.SaveChangesAsync();
}
}
}
Timer class:
namespace EmailNews.Services
{
#region snippet1
internal class TimedHostedService : IHostedService, IDisposable
{
private readonly ILogger _logger;
private Timer _timer;
public TimedHostedService(IServiceProvider services, ILogger<TimedHostedService> logger)
{
Services = services;
_logger = logger;
}
public IServiceProvider Services { get; }
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Background Service is starting.");
_timer = new Timer(DoWork, null, TimeSpan.Zero,
TimeSpan.FromMinutes(1));
return Task.CompletedTask;
}
private void DoWork(object state)
{
using (var scope = Services.CreateScope())
{
var scopedProcessingService =
scope.ServiceProvider
.GetRequiredService<IScopedProcessingService>();
scopedProcessingService.DoWork();
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Background Service is stopping.");
_timer?.Change(Timeout.Infinite, 0);
return Task.CompletedTask;
}
public void Dispose()
{
_timer?.Dispose();
}
}
#endregion
}
Startup:
services.AddHostedService<TimedHostedService>();
services.AddScoped<IScopedProcessingService, ScopedProcessingService>();
It seems everything is done as in the example, but nothing is added to the database, which is not so?
This is a rather interesting question, that boils down to "How do you correctly handle an async timer callback?"
The immediate problem is that SaveChangesAsync isn't getting awaited. The DbContext almost certainly gets disposed before SaveChangesAsync has a chance to run. To await it, DoWork must become an async Task method (never async void) :
internal interface IScheduledTask
{
Task DoWorkAsync();
}
internal class MailTask : IScheduledTask
{
private readonly ApplicationDbContext _context;
public MailTask(ApplicationDbContext context)
{
_context = context;
}
public async Task DoWorkAsync()
{
var mail = new Mail
{ Date = DateTime.Now,
Note = "lala",
Tema = "lala",
Email = "lala" };
_context.Add(mail);
await _context.SaveChangesAsync();
}
}
The problem now is how to call DoWorkAsync from the timer callback. If we just call it without awaiting, we'll get the same problem we had in the first place. A timer callback can't handle methods that return Task. We can't make it async void either, because this would result in the same problem - the method will return before any async operation has a chance to finish.
David Fowler explains how to properly handle asynchronous timer callbacks in the Timer Callbacks section of his Async Guidance
article :
private readonly Timer _timer;
private readonly HttpClient _client;
public Pinger(HttpClient client)
{
_client = new HttpClient();
_timer = new Timer(Heartbeat, null, 1000, 1000);
}
public void Heartbeat(object state)
{
// Discard the result
_ = DoAsyncPing();
}
private async Task DoAsyncPing()
{
await _client.GetAsync("http://mybackend/api/ping");
}
The actual method should be async Task but the returned task only has to be assigned, not awaited, in order for it to work properly.
Applying this to the question leads to something like this :
public Task StartAsync(CancellationToken cancellationToken)
{
...
_timer = new Timer(HeartBeat, null, TimeSpan.Zero,
TimeSpan.FromMinutes(1));
return Task.CompletedTask;
}
private void Heartbeat(object state)
{
_ = DoWorkAsync();
}
private async Task DoWorkAsync()
{
using (var scope = Services.CreateScope())
{
var schedTask = scope.ServiceProvider
.GetRequiredService<IScheduledTask>();
await schedTask.DoWorkAsync();
}
}
David Fowler explains why async void is ALWAY BAD in ASP.NET Core - it's not only that async actions won't be awaited, exceptions will crash the application.
He also explains why we can't use Timer(async state=>DoWorkAsync(state)) - that's an async void delegate.

.Net Core Queue Background Tasks

Slender answered my original question about what happens to fire and forget, after the HTTP Response is sent, but Now I'm left with the question how to properly queue background tasks
EDIT
As we all know Async void is generally bad, except for in the case when it comes to event handlers, I would like to execute some background logic without have to have the client wait. My original Idea was to use Fire and Forget
Say I have an event:
public event EventHandler LongRunningTask;
And then someone subscribes a fire and forget task:
LongRunningTask += async(s, e) => { await LongNetworkOperation;};
the web api method is call:
[HttpGet]
public async IActionResult GetTask()
{
LongRunningTask?.Invoke(this, EventArgs.Empty);
return Ok();
}
But If I do this my long running task isn't guaranteed to finish, How can I handle running background task without affect the time the time it take to make my request (e.g I don't want to wait for the task to finish first)?
.NET Core 2.1 has an IHostedService, which will safely run tasks in the background. I've found an example in the documentation for QueuedHostedService which I've modified to use the BackgroundService.
public class QueuedHostedService : BackgroundService
{
private Task _backgroundTask;
private readonly ILogger _logger;
public QueuedHostedService(IBackgroundTaskQueue taskQueue, ILoggerFactory loggerFactory)
{
TaskQueue = taskQueue;
_logger = loggerFactory.CreateLogger<QueuedHostedService>();
}
public IBackgroundTaskQueue TaskQueue { get; }
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
while (false == stoppingToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(stoppingToken);
try
{
await workItem(stoppingToken);
}
catch (Exception ex)
{
this._logger.LogError(ex, $"Error occurred executing {nameof(workItem)}.");
}
}
}
}
public interface IBackgroundTaskQueue
{
void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem);
Task<Func<CancellationToken, Task>> DequeueAsync(
CancellationToken cancellationToken);
}
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private ConcurrentQueue<Func<CancellationToken, Task>> _workItems =
new ConcurrentQueue<Func<CancellationToken, Task>>();
private SemaphoreSlim _signal = new SemaphoreSlim(0);
public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
{
if (workItem == null)
{
throw new ArgumentNullException(nameof(workItem));
}
_workItems.Enqueue(workItem);
_signal.Release();
}
public async Task<Func<CancellationToken, Task>> DequeueAsync( CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_workItems.TryDequeue(out var workItem);
return workItem;
}
}
Now we can safely queue up tasks in the background without affecting the time it takes to respond to a request.
Just wanted to add some additional notes to #johnny5 answer. Right now you can use https://devblogs.microsoft.com/dotnet/an-introduction-to-system-threading-channels/ instead of ConcurrentQueue with Semaphore.
The code will be something like this:
public class HostedService: BackgroundService
{
private readonly ILogger _logger;
private readonly ChannelReader<Stream> _channel;
public HostedService(
ILogger logger,
ChannelReader<Stream> channel)
{
_logger = logger;
_channel = channel;
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
await foreach (var item in _channel.ReadAllAsync(cancellationToken))
{
try
{
// do your work with data
}
catch (Exception e)
{
_logger.Error(e, "An unhandled exception occured");
}
}
}
}
[ApiController]
[Route("api/data/upload")]
public class UploadController : ControllerBase
{
private readonly ChannelWriter<Stream> _channel;
public UploadController (
ChannelWriter<Stream> channel)
{
_channel = channel;
}
public async Task<IActionResult> Upload([FromForm] FileInfo fileInfo)
{
var ms = new MemoryStream();
await fileInfo.FormFile.CopyToAsync(ms);
await _channel.WriteAsync(ms);
return Ok();
}
}

Categories