We Keep Coding

.NET5 Background Service Concurrency

I have a background service that will be started when the application performing startup. The background service will start to create multiple tasks based on how many workers are set. As I do various trials and monitor the open connection on DB. The open connection is always the same value as the worker I set. Let say I set 32 workers, then the connection will be always 32 open connections shown as I use query to check it. FYI I am using Postgres as the DB server. In order to check the open connection, I use the query below to check the connection when the application is running.
select * from pg_stat_activity where application_name = 'myapplication';
Below is the background service code.
public class MessagingService : BackgroundService {
private int worker = 32;
protected override async Task ExecuteAsync(CancellationToken cancellationToken) {
var tasks = new List<Task>();
for (int i=0; i<worker; i++) {
tasks.Add(DoJob(cancellationToken));
}
while (!cancellationToken.IsCancellationRequested) {
try {
var completed = await Task.WhenAny(tasks);
tasks.Remove(completed);
} catch (Exception) {
await Task.Delay(1000, cancellationToken);
}
if (!cancellationToken.IsCancellationRequested) {
tasks.Add(DoJob(cancellationToken));
}
}
}
private async Task DoJob(CancellationToken cancellationToken) {
using (var scope = _services.CreateScope()) {
var service = scope.ServiceProvider
.GetRequiredService<MessageService>();
try {
//do select and update query on db if null return false otherwise send mail
if (!await service.Run(cancellationToken)) {
await Task.Delay(1000, cancellationToken);
}
} catch (Exception) {
await Task.Delay(1000, cancellationToken);
}
}
}
}
The workflow is not right as it will keep creating the task and leave the connection open and idle. Also, the CPU and memory usage are high when running those tasks. How can I achieve like when there is no record found on DB only keep 1 worker running at the moment? If a record or more is found it will keep increasing until the preset max worker then decreasing the worker when the record is less than the max worker. If this question is too vague or opinion-based then please let me know and I will try my best to make it as specific as possible.
Update Purpose
The purpose of this service is to perform email delivery. There is another API that will be used to create a scheduled job. Once the job is added to the DB, this service will do the email delivery at the scheduled time. Eg, 5k schedule jobs are added to the DB and the scheduled time to perform the job is '2021-12-31 08:00:00' and the time when creating the scheduled job is 2021-12-31 00:00:00'. The service will keep on looping from 00:00:00 until 08:00:00 with 32 workers running at the same time then just start to do the email delivery. How can I improve it to more efficiency like normally when there is no job scheduled only 1 worker is running. When it checked there is 5k scheduled job it will fully utilise all the worker. After 5k job is completed, it will back to 1 workers.

My suggestion is to spare yourself from the burden of manually creating and maintaining worker tasks, by using an ActionBlock<T> from the TPL Dataflow library. This component is a combination of an input queue and an Action<T> delegate. You specify the delegate in its constructor, and you feed it with messages with its Post method. The component invokes the delegate for each message it receives, with the specified degree of parallelism. When there are no more messages to send, you notify it by invoking its Complete method, and then await its Completion so that you know that all work that was delegated to it has completed.
Below is a rough demonstration if how you could use this component:
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
var processor = new ActionBlock<Job>(async job =>
{
await ProcessJob(job);
await MarkJobAsCompleted(job);
}, new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 32
});
try
{
while (true)
{
Task delayTask = Task.Delay(TimeSpan.FromSeconds(60), cancellationToken);
Job[] jobs = await FetchReadyToProcessJobs();
foreach (var job in jobs)
{
await MarkJobAsPending(job);
processor.Post(job);
}
await delayTask; // Will throw when the token is canceled
}
}
finally
{
processor.Complete();
await processor.Completion;
}
}
The FetchReadyToProcessJobs method is supposed to connect to the database, and fetch all the jobs whose time has come to be processed. In the above example this method is invoked every 60 seconds. The Task.Delay is created before invoking the method, and awaited after the returned jobs have been posted to the ActionBlock<T>. This way the interval between invocations will be stable and consistent.

Is there a better, maybe more reliable pattern than "fire and forget" to handle variable number of asynchronous tasks concurrently?

I have following code:
while (!cancellationToken.IsCancellationRequested)
{
var connection = await listener.AcceptAsync(cancellationToken);
HandleConnectionAsync(connection, cancellationToken)
.FireAndForget(HandleException);
}
The FireAndForget is an extension method:
public static async void FireAndForget(this ValueTask task, Action<Exception> exceptionHandler)
{
try
{
await task.ConfigureAwait(false);
}
catch (Exception e)
{
exceptionHandler.Invoke(e);
}
}
The while loop is the server lifecycle. When new connection is accepted then it starts some "background task" so it can handle this new connection and then while loop goes back to accepting new connections without awaiting anything - pausing the lifecycle.
I cannot await HandleConnectionAsync (pause the lifecycle) here, because I want to immediately accept another connection (if there is one) and be able to handle multiple connections concurrently. HandleConnectionAsync is I/O bound and handles one connection at time until closed (task completes after some time).
The connections have to be handled separately - I don't want to have a situation when some error while handling one connection have any influence on other connections.
The "fire and forget" solution I have here works, but the general rule is to always await asynchronous methods and never use async void.
It seems like I've broken the rules, so is there a better, maybe more reliable way to handle variable (number of tasks varies in time) number of asynchronous I/O bound tasks concurrently in a situation described here?
More information:
Each call to AcceptAsync allocates system resources even before returning the connection and I want to avoid that whenever possible (the connection may not be returned for hours (code may "await" for hours) - until some external client decides to connect to my server). It is better to assume that this is the method I don't want to be called concurrently/in parallel - just one AcceptAsync at time is enough
Please take into account that I can have millions of clients per day connecting and disconnecting to my server and server (while loop) can work for many many days
I don't know how many connections I will need to handle at a specific time
I do know the maximum number of connections my program will be able to handle concurrently
If I hit the maximum number of connections limit then AcceptAsync won't return new connection until some other active connection closes, so I don't need to worry about that, but any solution based on this limit have to take into account that the active connections may be closed and I still need to handle new connections - number of connections varies over time. "fire and forget" have no issues with that
The code for HandleConnectionAsync is not relevant - it just handles one connection at time until closed (task completes after some time) and is I/O bound (HandleConnectionAsync handles one connection at time, but of course we can start multiple HandleConnectionAsync tasks to handle multiple connections concurrently - which is what I did with "fire and forget")

I'm assuming that changing to something like SignalR isn't an acceptable solution. That would be my first recommendation.
Custom server sockets is a scenario where some kind of "fire and forget" is acceptable. I'm considering adding a "task manager" kind of type to AsyncEx to make this kind of solution easier, but haven't done it yet.
The bottom line is that you need to manage your list of connections yourself. The "connection" object can include a Task that represents the handling loop; that's fine. It's also useful (especially for debugging or management purposes) to have other properties on there as well, such as the remote IP.
So I would approach it something like this:
private readonly object _mutex = new object();
private readonly List<State> _connections = new List<State>();
private void Add(State state)
{
lock (_mutex)
_connections.Add(state);
}
private void Remove(State state)
{
lock (_mutex)
_connections.Remove(state);
}
public async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
var connection = await listener.AcceptAsync(cancellationToken);
Add(new State(this, connection));
}
}
private sealed class State
{
private readonly Parent _parent;
public State(Parent parent, Connection connection, CancellationToken cancellationToken)
{
_parent = parent;
Task = ExecuteAsync(connection, cancellationToken);
}
private static async Task ExecuteAsync(Connection connection, CancellationToken cancellationToken)
{
try { await HandleConnectionAsync(connection, cancellationToken); }
finally { _parent.Remove(this); }
}
public Task Task { get; }
// other properties as desired, e.g., RemoteAddress
}
You now have a collection of connections. You can either ignore the tasks in the State objects (as the code above is doing), which is just like fire-and-forget. Or you can await them all at some point. E.g.:
public async Task RunAsync(CancellationToken cancellationToken)
{
try
{
while (true)
{
var connection = await listener.AcceptAsync(cancellationToken);
Add(new State(this, connection));
}
}
catch (OperationCanceledException)
{
// Wait for all connections to cancel.
// I'm not really sure why you would *want* to do this, though.
List<State> connections;
lock (_mutex) { connections = _connections.ToList(); }
await Task.WhenAll(connections.Select(x => x.Task));
}
}
Then it's easy to extend the State object so you can do things that are sometimes useful for a server app to do, e.g.:
List all remote addresses this server has connections to.
Wait until a specific connection is done.
...
Notes:
Use one pattern for cancellation. Passing the token will result in an OperationCanceledException, which is the normal cancellation pattern. The code also was formerly doing a while (!IsCancellationRequested), resulting in a successful completion on cancellation, which is not the normal cancellation pattern. So I removed that so the code is no longer using two cancellation patterns.
When working with raw sockets, in the general case, you need to be constantly reading (even when you're writing) and periodically writing (even if you have no data to send). So your HandleConnectionAsync should be starting an asynchronous reader and writer and then using Task.WhenAll.
I removed the call to HandleException because (probably) whatever it does should be handled by State.ExecuteAsync. It's not hard to add it back in if necessary.

If there is a limit to the maximum number of allowed concurrent tasks, you should use SemaphoreSlim:
int allowedConcurrent = //..
var semaphore = new SemaphoreSlim(allowedConcurrent);
var tasks = new List<Task>();
while (!cancellationToken.IsCancellationRequested)
{
Func<Task> func = async () =>
{
var connection = await listener.AcceptAsync(cancellationToken);
await HandleConnectionAsync(connection, cancellationToken);
semaphore.Release();
};
await semaphore.WaitAsync(); // Will return immediately if the number of concurrent tasks does not exceed allowed
tasks.Add(func());
}
await Task.WhenAll(tasks);
This will accumulate the tasks into a list, then Task.WhenAll can wait for them all to complete.

First things first:
Don't do async void...
Then you can implement a producer/consumer pattern for this, the below pseudocode is just to guide, you need to make sure your Consumer is a Singleton in your app
public class Data
{
public Uri Url { get; set; }
}
public class Producer
{
private Consumer _consumer = new Consumer();
public void DoStuff()
{
var data = new Data();
_consumer.Enqueue(data);
}
}
public class Consumer
{
private readonly List<Data> _toDo = new List<Data>();
private bool _stop = false;
public Consumer()
{
Task.Factory.StartNew(Loop);
}
private async Task Loop()
{
while (!_stop)
{
Data toDo = null;
lock (_toDo)
{
if (_toDo.Any())
{
toDo = _toDo.First();
_toDo.RemoveAt(0);
}
}
if (toDo != null)
{
await DoSomething(toDo);
}
Thread.Sleep(TimeSpan.FromSeconds(1));
}
}
private async Task DoSomething(Data toDo)
{
// YOUR ASYNC STUFF HERE
}
public void Enqueue(Data data)
{
lock (_toDo)
{
_toDo.Add(data);
}
}
}
So your calling method produces what you need to do the background task and the consumer performs that, that's another fire and forget.
You should consider too what happens if something goes wrong at an application level, should you store the Data in the Consumer.Enqueue() so if the app starts again can do the missing job...
Hope this helps

Contentious paralleled work between two collections of objects

I am trying to simulate work between two collections asynchronously and in parallel, I have a ConcurrentQueue of customers and a collection of workers. I need the workers to take a Customer from the queue perform work on the customer and once done take another customer right away.
I decided I'd use an event-based paradigm where the collection of workers would perform an action on a customer; who holds an event handler that fires off when the customer is done; which would hopefully fire off the DoWork Method once again, that way I can parallelize the workers to take customers from the queue. But I can't figure out how to pass a customer into DoWork in OnCustomerFinished()! The worker shouldn't depend on a queue of customers obviously
public class Worker
{
public async Task DoWork(ConcurrentQueue<Customer> cust)
{
await Task.Run(() =>
{
if (cust.TryDequeue(out Customer temp))
{
Task.Delay(5000);
temp.IsDone = true;
}
});
}
public void OnCustomerFinished()
{
// This is where I'm stuck
DoWork(~HOW TO PASS THE QUEUE OF CUSTOMER HERE?~);
}
}
// Edit - This is the Customer Class
public class Customer
{
private bool _isDone = false;
public EventHandler<EventArgs> CustomerFinished;
public bool IsDone
{
private get { return _isDone; }
set
{
_isDone = value;
if (_isDone)
{
OnCustomerFinished();
}
}
}
protected virtual void OnCustomerFinished()
{
if (CustomerFinished != null)
{
CustomerFinished(this, EventArgs.Empty);
}
}
}

.NET already has pub/sub and worker mechanisms in the form of DataFlow blocks and lately, Channels.
Dataflow
Dataflow blocks from the System.Threading.Tasks.Dataflow namespace are the "old" way (2012 and later) of building workers and pipelines of workers. Each block has an input and/or output buffer. Each message posted to the block is processed by one or more tasks in the background. For blocks with outputs, the output of each iteration is stored in the output buffer.
Blocks can be combined into pipelines similar to a CMD or Powershell pipeline, with each block running on its own task(s).
In the simplest case an ActionBlock can be used as a worker:
void ProcessCustomer(Customer customer)
{
....
}
var block =new ActionBlock<Customer>(cust=>ProcessCustomer(cust));
That's it. There's no need to manually dequeue or poll.
The producer method can start sending customer instances to the block. Each of them will be processed in the background, in the order they were posted :
foreach(var customer in bigCustomerList)
{
block.Post(customer);
}
When done, eg when the application terminates, the producer only needs to call Complete() on the block and wait for any remaining entries to complete.
block.Complete();
await block.Completion;
Blocks can work with asynchronous methods too.
Channels
Channels are a new mechanism, built into .NET Core 3 and available as a NuGet in previous .NET Framework and .NET Core version. The producer writes to a channel using a ChannelWriter and the consumer reads from the channel using a ChannelReader. This may seem a bit strange until you realize it allows some powerful patterns.
The producer could be something like this, eg a producer that "produces" all customers in a list with a 0.5 sec delay :
ChannelReader<Customer> Producer(IEnumerable<Customer> customers,CancellationToken token=default)
{
//Create a channel that can buffer an infinite number of entries
var channel=Channel.CreateUnbounded();
var writer=channel.Writer;
//Start a background task to produce the data
_ = Task.Run(async ()=>{
foreach(var customer in customers)
{
//Exit gracefully in case of cancellation
if (token.IsCancellationRequested)
{
return;
}
await writer.WriteAsync(customer,token);
await Task.Delay(500);
}
},token)
//Ensure we complete the writer no matter what
.ContinueWith(t=>writer.Complete(t.Exception);
return channel.Reader;
}
That's a bit more involved but notice that the only thing the function needs to return is the ChannelReader. The cancellation token is useful for terminating the producer early, eg after a timeout or if the application closes.
When the writer completes, all the channel's readers will also complete.
The consumer only needs that ChannelReader to work :
async Task Consumer(ChannelReader<Customer> reader,CancellationToken token=default)
{
while(await reader.WaitToReadAsync(token))
{
while(reader.TryRead(out var customer))
{
//Process the customer
}
}
}
Should the writer complete, WaitToReadAsync will return false and the loop will exit.
In .NET Core 3 the ChannelReader supports IAsyncEnumerable through the ReadAllAsync method, making the code even simpler :
async Task Consumer(ChannelReader<Customer> reader,CancellationToken token=default)
{
await foreach(var customer in reader.ReadAllAsync(token))
{
//Process the customer
}
}
The reader created by the producer can be passed directly to the consumer :
var customers=new []{......}
var reader=Producer(customers);
await Consumer(reader);
Intermediate steps can read from a previous channel reader and publish data to the next, eg an order generator :
ChannelReader<Order> ConsumerOrders(ChannelReader<Customer> reader,CancellationToken token=default)
{
var channel=Channel.CreateUnbounded();
var writer=channel.Writer;
//Start a background task to produce the data
_ = Task.Run(async ()=>{
await foreach(var customer in reader.ReadAllAsync(token))
{
//Somehow create an order for the customer
var order=new Order(...);
await writer.WriteAsync(order,token);
}
},token)
//Ensure we complete the writer no matter what
.ContinueWith(t=>writer.Complete(t.Exception);
return channel.Reader;
}
Again, all we need to do is pass the readers from one method to the next
var customers=new []{......}
var customerReader=Producer(customers);
var orderReader=CustomerOrders(customerReader);
await ConsumeOrders(orderReader);

Task.Run() schedules tasks randomly / erratically under high loads

As I am working on processing bulk of emails, I have used the Task method to process those emails asynchronously without affecting the primary work. (Basically sending email functionality should work on different thread than primary thread.) Imagine you're processing more than 1K email per 30 seconds in Windows Service.
The issue I am facing is-- many time the Task method is not executed, it completely behave randomly. Technically, it schedules the task randomly. Sometime I receive the call in SendEmail method and sometime not. I have tried both the approaches as mentioned below.
Method 1
public void ProcessMails()
{
Task.Run(() => SendEmail(emailModel));
}
Method 2
public async void ProcessMails()
{
// here the SendEmail method is awaitable, but I have not used 'await' because
// I need non-blocking operation on main thread.
SendEmail(emailModel));
}
Would anybody please let me know what could be the problem OR I am missing anything here?

As has been noted it seems as though you're running out resources to schedule tasks that ultimately send your emails. Right now the sample code tries to force feed all the work that needs to get scheduled immediatly.
The other answer provided suggects using a blocking collection but I think there is a cleaner easier way. This sample should at least give you the right idea.
using System;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
namespace ClassLibrary1
{
public class MailHandler
{
private EmailLibrary emailLibrary = new EmailLibrary();
private ExecutionDataflowBlockOptions options = new ExecutionDataflowBlockOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount };
private ActionBlock<string> messageHandler;
public MailHandler() => messageHandler = new ActionBlock<string>(msg => DelegateSendEmail(msg), options);
public Task ProcessMail(string message) => messageHandler.SendAsync(message);
private Task DelegateSendEmail(string message) => emailLibrary.SendEmail(message);
}
public class EmailLibrary
{
public Task SendEmail(string message) => Task.Delay(1000);
}
}

Given the fairly high frequency of sending email, it is likely that you are scheduling too many Tasks for the Scheduler.
In Method 1, calling Task.Run will create a new task each time, each of which needs to be scheduled on a thread. It is quite likely that you are exhausting your thread pool by doing this.
Although Method 2 will be less Task hungry, even with unawaited Task invocation (fire and forget), the completion of the continuation after the async method will still need to be scheduled on the Threadpool, which will adversely affect your system.
Instead of unawaited Tasks or Task.Run, and since you are a Windows Service, I would instead have a long-running background thread dedicated to sending emails. This thread can work independently to your primary work, and emails can be scheduled to this thread via a queue.
If a single mail sending thread is insufficient to keep pace with the mails, you can extend the number of EmailSender threads, but constrain this to a reasonable, finite number).
You should explore other optimizations too, which again will improve the throughput of your email sender e.g.
Can the email senders keep long lived connections to the mail server?
Does the mail server accept batches of email?
Here's an example using BlockingCollection with a backing ConcurrentQueue of your email message Model.
Creating a queue which is shared between the producer "PrimaryWork" thread and the "EmailConsumer" thread (obviously, if you have an IoC container, it's best registered there)
Enqueuing mails on the primary work thread
The consumer EmailSender runs a loop on the blocking collection queue until CompleteAdding is called
I've used a TaskCompletionSource to provide a Task which will complete once all messages have been sent, i.e. so that graceful exit is possible without losing emails still in the queue.
public class PrimaryWork
{
private readonly BlockingCollection<EmailModel> _enqueuer;
public PrimaryWork(BlockingCollection<EmailModel> enqueuer)
{
_enqueuer = enqueuer;
}
public void DoWork()
{
// ... do your work
for (var i = 0; i < 100; i++)
{
EnqueueEmail(new EmailModel {
To = $"recipient{i}#foo.com",
Message = $"Message {i}" });
}
}
// i.e. Queue work for the email sender
private void EnqueueEmail(EmailModel message)
{
_enqueuer.Add(message);
}
}
public class EmailSender
{
private readonly BlockingCollection<EmailModel> _mailQueue;
private readonly TaskCompletionSource<string> _tcsIsCompleted
= new TaskCompletionSource<string>();
public EmailSender(BlockingCollection<EmailModel> mailQueue)
{
_mailQueue = mailQueue;
}
public void Start()
{
Task.Run(() =>
{
try
{
while (!_mailQueue.IsCompleted)
{
var nextMessage = _mailQueue.Take();
SendEmail(nextMessage).Wait();
}
_tcsIsCompleted.SetResult("ok");
}
catch (Exception)
{
_tcsIsCompleted.SetResult("fail");
}
});
}
public async Task Stop()
{
_mailQueue.CompleteAdding();
await _tcsIsCompleted.Task;
}
private async Task SendEmail(EmailModel message)
{
// IO bound work to the email server goes here ...
}
}
Example of bootstrapping and starting the above producer / consumer classes:
public static async Task Main(string[] args)
{
var theQueue = new BlockingCollection<EmailModel>(new ConcurrentQueue<EmailModel>());
var primaryWork = new PrimaryWork(theQueue);
var mailSender = new EmailSender(theQueue);
mailSender.Start();
primaryWork.DoWork();
// Wait for all mails to be sent
await mailSender.Stop();
}
I've put a complete sample up on Bitbucket here
Other Notes
The blocking collection (and the backing ConcurrentQueue) are thread safe, so you can concurrently use more than one many producer and consumer thread.
As per above, batching is encouraged, and asynchronous parallelism is possible (Since each mail sender uses a thread, Task.WaitAll(tasks) will wait for a batch of tasks). A totally asynchronous sender could obviously use await Task.WhenAll(tasks).
As per comments below, I believe the nature of your system (i.e. Windows Service, with 2k messages / minute) warrants at least one dedicated thread for email sending, despite emailing likely being inherently IO bound.

Producer/consumer pattern in .NET with the ability to wait for submitted tasks/jobs to complete

I am trying to get my head around the following design, but fail to get a clear picture.
I have a number of producers submitting tasks/jobs to a queue. A consumer/worker would then pick these up and complete on these. For now, there is only one consumer/worker.
So far, this sounds like the standard producer/consumer pattern which could be done with a BlockingCollection.
However, some producers might want to submit a task/job and be able to wait for its completion (or submit multiple tasks/jobs and wait for some or all of them, etc.), while other producers would just "fire&forget" their tasks/jobs.
(Note that this is not waiting for the queue to be empty, but waiting for a particular task/job).
How would this be done? In all examples I have seen, producers just post data to the queue using BlockingQueue.Add().
Any help would be highly appreciated.

A common approach is to wrap your work operations using a TaskCompletionSource whose Task can be returned to the caller and awaited on for completion.
public class ProducerConsumerQueue
{
private readonly BlockingCollection<Action> queue = new BlockingCollection<Action>();
public Task Produce(Action work)
{
var tcs = new TaskCompletionSource<bool>();
Action action = () =>
{
try
{
work();
tcs.SetResult(true);
}
catch (Exception ex)
{
tcs.SetException(ex);
}
};
queue.Add(action);
return tcs.Task;
}
public void RunConsumer(CancellationToken token)
{
while (true)
{
token.ThrowIfCancellationRequested();
var action = queue.Take(token);
action();
}
}
}
That said, you should consider leveraging the task infrastructure provided by TPL itself, rather than coming up with your own structures. If your only requirement is having a bounded number of consumers, you could use a LimitedConcurrencyLevelTaskScheduler.