We Keep Coding

.NET 5 BackgroundService in Kubernetes Doesn't Exit

I'm able to successfully run a .NET 5 Console Application with a BackgroundService in an Azure Kubernetes cluster on Ubuntu 18.04. In fact, the BackgroundService is all that really runs: just grabs messages from a queue, executes some actions, then terminates when Kubernetes tells it to stop, or the occasional exception.
It's this last scenario which is giving me problems. When the BackgroundService hits an unrecoverable exception, I'd like the container to stop (complete, or whatever state will cause Kubernetes to either restart or destroy/recreate the container).
Unfortunately, any time an exception is encountered, the BackgroundService appears to hit the StopAsync() function (from what I can see in the logs and console output), but the container stays in a running state and never restarts. My Main() is as appears below:
public static async Task Main(string[] args)
{
// Build service host and execute.
var host = CreateHostBuilder(args)
.UseConsoleLifetime()
.Build();
// Attach application event handlers.
AppDomain.CurrentDomain.ProcessExit += OnProcessExit;
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);
try
{
Console.WriteLine("Beginning WebSec.Scanner.");
await host.StartAsync();
await host.WaitForShutdownAsync();
Console.WriteLine("WebSec.Scanner has completed.");
}
finally
{
Console.WriteLine("Cleaning up...");
// Ensure host is properly disposed.
if (host is IAsyncDisposable ad)
{
await ad.DisposeAsync();
}
else if (host is IDisposable d)
{
d.Dispose();
}
}
}
If relevant, those event handlers for ProcessExit and UnhandledException exist to flush the AppInsights telemetry channel (maybe that's blocking it?):
private static void OnProcessExit(object sender, EventArgs e)
{
// Ensure AppInsights logs are submitted upstream.
Console.WriteLine("Flushing logs to AppInsights");
TelemetryChannel.Flush();
}
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
var thrownException = (Exception)e.ExceptionObject;
Console.WriteLine("Unhandled exception thrown: {0}", thrownException.Message);
// Ensure AppInsights logs are submitted upstream.
Console.WriteLine("Flushing logs to AppInsights");
TelemetryChannel.Flush();
}
I am only overriding ExecuteAsync() in the BackgroundService:
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
this.logger.LogInformation(
"Service started.");
try
{
// Loop until the service is terminated.
while (!stoppingToken.IsCancellationRequested)
{
// Do some work...
}
}
catch (Exception ex)
{
this.logger.LogWarning(
ex,
"Terminating due to exception.");
}
this.logger.LogInformation(
"Service ending.",
}
My Dockerfile is simple and has this line to run the service:
ENTRYPOINT ["dotnet", "MyService.dll"]
Am I missing something obvious? I feel like there's something about running this as a Linux container that I'm forgetting in order to make this run properly.
Thank you!

Here is a full example of how to use IHostApplicationLifetime.StopApplication().
void Main()
{
var host = Host.CreateDefaultBuilder()
.ConfigureServices((context, services) =>
{
services.AddHostedService<MyService>();
})
.Build();
Console.WriteLine("Starting service");
host.Run();
Console.WriteLine("Ended service");
}
// You can define other methods, fields, classes and namespaces here
public class MyService : BackgroundService
{
private readonly IHostApplicationLifetime _lifetime;
private readonly Random _rnd = new Random();
public MyService(IHostApplicationLifetime lifetime)
{
_lifetime = lifetime;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
while (true)
{
stoppingToken.ThrowIfCancellationRequested();
var nextNumber = _rnd.Next(10);
if (nextNumber < 8)
{
Console.WriteLine($"We have number {nextNumber}");
}
else
{
throw new Exception("Number too high");
}
await Task.Delay(1000);
}
}
// If the application is shutting down, ignore it
catch (OperationCanceledException e) when (e.CancellationToken == stoppingToken)
{
Console.WriteLine("Application is shutting itself down");
}
// Otherwise, we have a real exception, so must ask the application
// to shut itself down.
catch (Exception e)
{
Console.WriteLine("Oh dear. We have an exception. Let's end the process.");
// Signal to the OS that this was an error condition by
// setting the exit code.
Environment.ExitCode = 1;
_lifetime.StopApplication();
}
}
}
Typical output from this program will look like:
Starting service
We have number 0
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: C:\Users\rowla\AppData\Local\Temp\LINQPad6\_spgznchd\shadow-1
We have number 2
Oh dear. We have an exception. Let's end the process.
info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...
Ended service

Forcing certain code to always run on the same thread

We have an old 3rd party system (let's call it Junksoft® 95) that we interface with via PowerShell (it exposes a COM object) and I'm in the process of wrapping it in a REST API (ASP.NET Framework 4.8 and WebAPI 2). I use the System.Management.Automation nuget package to create a PowerShell in which I instantiate Junksoft's COM API as a dynamic object that I then use:
//I'm omitting some exception handling and maintenance code for brevity
powerShell = System.Management.Automation.PowerShell.Create();
powerShell.AddScript("Add-Type -Path C:\Path\To\Junksoft\Scripting.dll");
powerShell.AddScript("New-Object Com.Junksoft.Scripting.ScriptingObject");
dynamic junksoftAPI = powerShell.Invoke()[0];
//Now we issue commands to junksoftAPI like this:
junksoftAPI.Login(user,pass);
int age = junksoftAPI.GetAgeByCustomerId(custId);
List<string> names = junksoftAPI.GetNames();
This works fine when I run all of this on the same thread (e.g. in a console application). However, for some reason this usually doesn't work when I put junksoftAPI into a System.Web.Caching.Cache and use it from different controllers in my web app. I say ususally because this actually works when ASP.NET happens to give the incoming call to the thread that junksoftAPI was created on. If it doesn't, Junksoft 95 gives me an error.
Is there any way for me to make sure that all interactions with junksoftAPI happen on the same thread?
Note that I don't want to turn the whole web application into a single-threaded application! The logic in the controllers and elswhere should happen like normal on different threads. It should only be the Junksoft interactions that happen on the Junksoft-specific thread, something like this:
[HttpGet]
public IHttpActionResult GetAge(...)
{
//finding customer ID in database...
...
int custAge = await Task.Run(() => {
//this should happen on the Junksoft-specific thread and not the next available thread
var cache = new System.Web.Caching.Cache();
var junksoftAPI = cache.Get(...); //This has previously been added to cache on the Junksoft-specific thread
return junksoftAPI.GetAgeByCustomerId(custId);
});
//prepare a response using custAge...
}

You can create your own singleton worker thread to achieve this. Here is the code which you can plug it into your web application.
public class JunkSoftRunner
{
private static JunkSoftRunner _instance;
//singleton pattern to restrict all the actions to be executed on a single thread only.
public static JunkSoftRunner Instance => _instance ?? (_instance = new JunkSoftRunner());
private readonly SemaphoreSlim _semaphore;
private readonly AutoResetEvent _newTaskRunSignal;
private TaskCompletionSource<object> _taskCompletionSource;
private Func<object> _func;
private JunkSoftRunner()
{
_semaphore = new SemaphoreSlim(1, 1);
_newTaskRunSignal = new AutoResetEvent(false);
var contextThread = new Thread(ThreadLooper)
{
Priority = ThreadPriority.Highest
};
contextThread.Start();
}
private void ThreadLooper()
{
while (true)
{
//wait till the next task signal is received.
_newTaskRunSignal.WaitOne();
//next task execution signal is received.
try
{
//try execute the task and get the result
var result = _func.Invoke();
//task executed successfully, set the result
_taskCompletionSource.SetResult(result);
}
catch (Exception ex)
{
//task execution threw an exception, set the exception and continue with the looper
_taskCompletionSource.SetException(ex);
}
}
}
public async Task<TResult> Run<TResult>(Func<TResult> func, CancellationToken cancellationToken = default(CancellationToken))
{
//allows only one thread to run at a time.
await _semaphore.WaitAsync(cancellationToken);
//thread has acquired the semaphore and entered
try
{
//create new task completion source to wait for func to get executed on the context thread
_taskCompletionSource = new TaskCompletionSource<object>();
//set the function to be executed by the context thread
_func = () => func();
//signal the waiting context thread that it is time to execute the task
_newTaskRunSignal.Set();
//wait and return the result till the task execution is finished on the context/looper thread.
return (TResult)await _taskCompletionSource.Task;
}
finally
{
//release the semaphore to allow other threads to acquire it.
_semaphore.Release();
}
}
}
Console Main Method for testing:
public class Program
{
//testing the junk soft runner
public static void Main()
{
//get the singleton instance
var softRunner = JunkSoftRunner.Instance;
//simulate web request on different threads
for (var i = 0; i < 10; i++)
{
var taskIndex = i;
//launch a web request on a new thread.
Task.Run(async () =>
{
Console.WriteLine($"Task{taskIndex} (ThreadID:'{Thread.CurrentThread.ManagedThreadId})' Launched");
return await softRunner.Run(() =>
{
Console.WriteLine($"->Task{taskIndex} Completed On '{Thread.CurrentThread.ManagedThreadId}' thread.");
return taskIndex;
});
});
}
}
}
Output:
Notice that, though the function was launched from the different threads, some portion of code got always executed always on the same context thread with ID: '5'.
But beware that, though all the web requests are executed on independent threads, they will eventually wait for some tasks to get executed on the singleton worker thread. This will eventually create a bottle neck in your web application. This is anyway your design limitation.

Here is how you could issue commands to the Junksoft API from a dedicated STA thread, using a BlockingCollection class:
public class JunksoftSTA : IDisposable
{
private readonly BlockingCollection<Action<Lazy<dynamic>>> _pump;
private readonly Thread _thread;
public JunksoftSTA()
{
_pump = new BlockingCollection<Action<Lazy<dynamic>>>();
_thread = new Thread(() =>
{
var lazyApi = new Lazy<dynamic>(() =>
{
var powerShell = System.Management.Automation.PowerShell.Create();
powerShell.AddScript("Add-Type -Path C:\Path\To\Junksoft.dll");
powerShell.AddScript("New-Object Com.Junksoft.ScriptingObject");
dynamic junksoftAPI = powerShell.Invoke()[0];
return junksoftAPI;
});
foreach (var action in _pump.GetConsumingEnumerable())
{
action(lazyApi);
}
});
_thread.SetApartmentState(ApartmentState.STA);
_thread.IsBackground = true;
_thread.Start();
}
public Task<T> CallAsync<T>(Func<dynamic, T> function)
{
var tcs = new TaskCompletionSource<T>(
TaskCreationOptions.RunContinuationsAsynchronously);
_pump.Add(lazyApi =>
{
try
{
var result = function(lazyApi.Value);
tcs.SetResult(result);
}
catch (Exception ex)
{
tcs.SetException(ex);
}
});
return tcs.Task;
}
public Task CallAsync(Action<dynamic> action)
{
return CallAsync<object>(api => { action(api); return null; });
}
public void Dispose() => _pump.CompleteAdding();
public void Join() => _thread.Join();
}
The purpose of using the Lazy class is for surfacing a possible exception during the construction of the dynamic object, by propagating it to the callers.
...exceptions are cached. That is, if the factory method throws an exception the first time a thread tries to access the Value property of the Lazy<T> object, the same exception is thrown on every subsequent attempt.
Usage example:
// A static field stored somewhere
public static readonly JunksoftSTA JunksoftStatic = new JunksoftSTA();
await JunksoftStatic.CallAsync(api => { api.Login("x", "y"); });
int age = await JunksoftStatic.CallAsync(api => api.GetAgeByCustomerId(custId));
In case you find that a single STA thread is not enough to serve all the requests in a timely manner, you could add more STA threads, all of them running the same code (private readonly Thread[] _threads; etc). The BlockingCollection class is thread-safe and can be consumed concurrently by any number of threads.

If you did not say that was a 3rd party tool, I would have asumed it is a GUI class. For practical reasons, it is a very bad idea to have multiple threads write to them. .NET enforces a strict "only the creating thread shall write" rule, from 2.0 onward.
WebServers in general and ASP.Net in particular use a pretty big thread pool. We are talking 10's to 100's of Threads per Core. That means it is really hard to nail any request down to a specific Thread. You might as well not try.
Again, looking at the GUI classes might be your best bet. You could basically make a single thread with the sole purpose of immitating a GUI's Event Queue. The Main/UI Thread of your average Windows Forms application, is responsible for creating every GUI class instance. It is kept alive by polling/processing the event queue. It ends onlyx when it receies a cancel command, via teh Event Queue. Dispatching just puts orders into that Queue, so we can avoid Cross-Threading issues.

dotnet core how to shutdown child process gracefully

I have a dotnet core 2.2 console app.
I hosted it as windows service.
Service starts up another dotnet core WebAPI.
The problem is, how do I gracefully shutdown WebAPI process when the the service is stopped?
Note: I don't want to use Kill() method.
Sample code:
public class MyService : IHostedService, IDisposable
{
private Timer _timer;
static Process webAPI;
public Task StartAsync(CancellationToken cancellationToken)
{
_timer = new Timer(
(e) => StartChildProcess(),
null,
TimeSpan.Zero,
TimeSpan.FromMinutes(1));
return Task.CompletedTask;
}
public void StartChildProcess()
{
try
{
webAPI = new Process();
webAPI.StartInfo.UseShellExecute = false;
webAPI.StartInfo.FileName = #"C:\Project\bin\Debug\netcoreapp2.2\publish\WebAPI.exe";
webAPI.Start();
}
catch (Exception e)
{
// Handle exception
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
// TODO: Add code to stop child process safely
_timer?.Change(Timeout.Infinite, 0);
return Task.CompletedTask;
}
public void Dispose()
{
_timer?.Dispose();
}
}

Technically you could simply call Process.Kill() in order to immediately shut down the process. However, a lot of the time that is not the way to go simply because the WebAPI might be in middle of important operations and you can't really tell when those actions may be happening and Process.Kill() is not really considered "graceful".
What would be most prudent to do is to tell the process that you would like for it to shut down at the earliest convenience and then allow for the WebAPI to clean things up before it exits itself. If you are desiging the WebAPI that is even better because that way you can decide on how to do this. Only calling Kill() when it is absolutely necessary.
You can do that multiple ways of course. Some that come to mind are Sockets that are periodically checked and sending a CTRL+C input to the WebAPI.
public Task StopAsync(CancellationToken cancellationToken)
{
// send request to shut down
// wait for process to exit and free its resources
process.WaitForExit();
process.Close();
process.Dispose();
_timer?.Change(Timeout.Infinite, 0);
return Task.CompletedTask;
}
Of course if this is Async then it wouldn't make sense to wait for it to exit inside of the method so you would simply wait or check if it has exited outside of this method.

There were a lot of threads regarding this issue on github, consider post 7426.
The solution is found here: StartAndStopDotNetCoreApp, the sample code of the program.cs is:
using System;
using System.IO;
using System.Diagnostics;
namespace StartAndStopDotNetCoreApp
{
class Program
{
static void Main(string[] args)
{
string projectPath = #"C:\source\repos\StartAndStopDotNetCoreApp\WebApplication";
string outputPath = #"C:\Temp\WebApplication";
Console.WriteLine("Starting the app...");
var process = new Process();
process.StartInfo.WorkingDirectory = projectPath;
process.StartInfo.FileName = "dotnet";
process.StartInfo.Arguments = $"publish -o {outputPath}";
process.StartInfo.UseShellExecute = false;
process.StartInfo.CreateNoWindow = true;
process.Start();
process.WaitForExit();
process.Close();
process.Dispose();
process = new Process();
process.StartInfo.WorkingDirectory = outputPath;
process.StartInfo.FileName = "dotnet";
process.StartInfo.Arguments = $"{projectPath.Split(#"\")[projectPath.Split(#"\").Length - 1]}.dll";
process.StartInfo.UseShellExecute = false;
process.StartInfo.CreateNoWindow = false;
process.StartInfo.RedirectStandardOutput = true;
process.Start();
Console.WriteLine("Press anything to stop...");
Console.Read();
process.Kill();
}
}
}
If this is not what you are looking for, search the mentioned thread for more ways, it offers plenty.

Perhaps it's possible to use the CloseMainWindow method of the System.Diagnostics.Process class? I stumbled upon it when I was looking for a way to send a WM_CLOSE message to another process from C#.
Edit:
CloseMainWindow does not seem to work if the main window is "blocked", for example by an open modal window or a dialog.
You might investigate a strategy for sending a WM_CLOSE or WM_DESTROY message to your app process explicitly. But I cannot guarantee if it will work as you want/expect. I haven't worked with that kind of Windows Messages functionality for a very long time.

How to properly implement Quartz.Net job lifecycle for long running processes?

I am using Quartz.Net to implement some asynchronous processing within the IIS worker process (IIS 8.5). One particular job may take more 10 minutes to run and does a lot of processing.
The following pieces of code illustrate how I am handling job life cycle.
Schedule definition
public class JobScheduler
{
private static IScheduler _quartzScheduler;
public static void Start()
{
_quartzScheduler = new StdSchedulerFactory().GetScheduler();
_quartzScheduler.JobFactory = new NinjectJobFactory();
ScheduleTheJob(_quartzScheduler);
_quartzScheduler.Context.Add("key", "scheduler");
_quartzScheduler.Start();
}
private static void ScheduleTheJob(IScheduler scheduler)
{
IJobDetail job = JobBuilder.Create<JobClass>().UsingJobData("JobKey", "JobValue").Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("JobTrigger", "JobGroup").UsingJobData("JobKey", "JobTrigger").StartNow()
.WithSimpleSchedule(x => x
// 30 minutes for refresh period
.WithIntervalInSeconds(ConfigService.JobRefreshPeriod)
.RepeatForever())
.Build();
scheduler.ScheduleJob(job, trigger);
}
public static void StopScheduler(bool waitForCompletion)
{
_quartzScheduler.Shutdown(waitForCompletion);
}
}
Application pool shutdown handling
public class ApplicationPoolService : IApplicationPoolService
{
public bool IsShuttingDown()
{
return System.Web.Hosting.HostingEnvironment.ShutdownReason != ApplicationShutdownReason.None;
}
public ApplicationShutdownReason GetShutdownReason()
{
return System.Web.Hosting.HostingEnvironment.ShutdownReason;
}
}
public class HostingEnvironmentRegisteredObject : IRegisteredObject
{
public void Stop(bool immediate)
{
if (immediate)
return;
JobScheduler.StopScheduler(waitForCompletion: true);
var logger = NinjectWebCommon.Kernel.Get<ILoggingService>();
var appPoolService = NinjectWebCommon.Kernel.Get<IApplicationPoolService>();
var reason = appPoolService.GetShutdownReason().ToString();
logger.Log(LogLevel.Info, $"HostingEnvironmentRegisteredObject.stop called with shutdown reason {reason}");
}
}
Global.asax.cs wiring up
protected void Application_Start()
{
JobScheduler.Start();
HostingEnvironment.RegisterObject(new HostingEnvironmentRegisteredObject());
}
protected void Application_Error()
{
Exception exception = Server.GetLastError();
Logger.Log(LogLevel.Fatal, exception, "Application global error");
}
protected void Application_End(object sender, EventArgs e)
{
// stopping is now triggered in HostingEnvironmentRegisteredObject
// JobScheduler.StopScheduler(false);
// get shutdown reason
var appPoolService = NinjectWebCommon.Kernel.Get<IApplicationPoolService>();
var reason = appPoolService.GetShutdownReason().ToString();
Logger.Log(LogLevel.Info, $"Application_End called with shutdown reason {reason}");
}
Job step description
if (ApplicationPoolService.IsShuttingDown())
{
Logger.Log(LogLevel.Info, "(RefreshEnvironmentImportingSystemData) Application pool is shutting down");
return;
}
// about 20-30 steps may be here
environments.ForEach(env =>
{
if (ApplicationPoolService.IsShuttingDown())
return;
// do heavy processing for about 2 minutes (worst case), typically some 10-20s
}
// one job step may allocate several hundreds of MB, so GC is called to reclaim some memory sooner
// it takes a few seconds (worst case)
GC.Collect();
GC.WaitForPendingFinalizers();
GC.WaitForFullGCComplete();
GC.Collect();
Initially, I was stopping the scheduler when Application_End was called, but I realized that was called when application pool was about to be killed, so I move it when application pool was notified that its shutdown has been started.
I have left application pool with its default value for Shutdown time limit (90 seconds).
Job is configured to not allow concurrent executions.
I want to achieve the following:
avoid forced killing of job during actual execution
minimize the time when two worker processes run in the same time (shutting down one in parallel with the one just started to handle new requests)
Question: did I manage correctly the scheduled jobs or can I make improvements?

How to create dependency injection for Azure Storage queues to process each message as it is provided

Trying to read from an Azure Queue to which some other service writes. If I use this in startup.cs
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=*;AccountKey=*;EndpointSuffix=*");
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue queue = queueClient.GetQueueReference("*");
queue.CreateIfNotExists();
var message= queue.GetMessage();
I can get the message in 'message' variable, but how to inject this in startup so that my processor class gets called with the message each time there is a new message in the queue. I tried to add a singleton by,
services.AddSinleton<ProcessorClassInterface>(x=> {return new ProcessorClass(queue)});
And then calling queue.GetMessage after every 1 second there.

This was solved by calling a function which uses multi-threading to poll the azure queues after the specified interval of time and fetch the messaged (with probably a set exponential back off time).
Approach 1:
To implement this in a webapp is a bit trickier, and I had to use a hack - call a function from the constructor to get the polling started.
In startup.cs (inside the configure function), register your service,
app.ApplicationServices.GetService<IQueueConsumer>();
In ConfigureServices Function, Configuring and creating an object of the polling queue class,
services.TryAddTransient<IQueueConsumer>(sp => this.GetQueueProcessor(sp));
And then, when the constructor is called to create the object, start polling a queue in a different thread.
public QueuePollingFunction(
IOptions<QueueOptions> queueOptions,
CloudQueue queue)
{
this.isEnabled = queueOptions.Value.IsEnabled;
this.StartPollingQueue(queue);
}
public override async Task<bool> ProcessMessageAsync(string message)
{
bool result = false;
try
{
var messageContent = JsonConvert.DeserializeObject<QueueEntity>(message);
result = true;
}
catch (Exception e)
{
Trace.TraceError(e.ToString());
}
return result;
}
private async Task StartPollingQueue(CloudQueue queue)
{
if (this.isEnabled)
{
Task pollQueue = Task.Factory.StartNew(() => Parallel.For(0, this.numberOfParallelTasks, work =>
{
this.Start(queue);
}));
}
}
private async Task Start(CloudQueue queue)
{
while (true)
{
try
{
CloudQueueMessage retrievedMessage = await queue.GetMessageAsync();
if (retrievedMessage != null)
{
// Fail Logic
if (retrievedMessage.DequeueCount > this.maxRetryLimit)
{
await queue.DeleteMessageAsync(retrievedMessage);
}
bool isPass = await this.ProcessMessageAsync(newChannelSettings);
if (isPass)
{
await queue.DeleteMessageAsync(retrievedMessage);
}
}
else
{
// If queue is empty, then the Task can sleep for sleepTime duration
await Task.Delay(this.sleepTime);
}
}
catch (Exception e)
{
Trace.TraceError(e.ToString());
}
}
}
Approach 2:
However, later had to move to the optimal approach, which is to use worker-roles and then uses Tasks to run a background thread to perform this task.