I'm able to successfully run a .NET 5 Console Application with a BackgroundService in an Azure Kubernetes cluster on Ubuntu 18.04. In fact, the BackgroundService is all that really runs: just grabs messages from a queue, executes some actions, then terminates when Kubernetes tells it to stop, or the occasional exception.
It's this last scenario which is giving me problems. When the BackgroundService hits an unrecoverable exception, I'd like the container to stop (complete, or whatever state will cause Kubernetes to either restart or destroy/recreate the container).
Unfortunately, any time an exception is encountered, the BackgroundService appears to hit the StopAsync() function (from what I can see in the logs and console output), but the container stays in a running state and never restarts. My Main() is as appears below:
public static async Task Main(string[] args)
{
// Build service host and execute.
var host = CreateHostBuilder(args)
.UseConsoleLifetime()
.Build();
// Attach application event handlers.
AppDomain.CurrentDomain.ProcessExit += OnProcessExit;
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(OnUnhandledException);
try
{
Console.WriteLine("Beginning WebSec.Scanner.");
await host.StartAsync();
await host.WaitForShutdownAsync();
Console.WriteLine("WebSec.Scanner has completed.");
}
finally
{
Console.WriteLine("Cleaning up...");
// Ensure host is properly disposed.
if (host is IAsyncDisposable ad)
{
await ad.DisposeAsync();
}
else if (host is IDisposable d)
{
d.Dispose();
}
}
}
If relevant, those event handlers for ProcessExit and UnhandledException exist to flush the AppInsights telemetry channel (maybe that's blocking it?):
private static void OnProcessExit(object sender, EventArgs e)
{
// Ensure AppInsights logs are submitted upstream.
Console.WriteLine("Flushing logs to AppInsights");
TelemetryChannel.Flush();
}
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
var thrownException = (Exception)e.ExceptionObject;
Console.WriteLine("Unhandled exception thrown: {0}", thrownException.Message);
// Ensure AppInsights logs are submitted upstream.
Console.WriteLine("Flushing logs to AppInsights");
TelemetryChannel.Flush();
}
I am only overriding ExecuteAsync() in the BackgroundService:
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
this.logger.LogInformation(
"Service started.");
try
{
// Loop until the service is terminated.
while (!stoppingToken.IsCancellationRequested)
{
// Do some work...
}
}
catch (Exception ex)
{
this.logger.LogWarning(
ex,
"Terminating due to exception.");
}
this.logger.LogInformation(
"Service ending.",
}
My Dockerfile is simple and has this line to run the service:
ENTRYPOINT ["dotnet", "MyService.dll"]
Am I missing something obvious? I feel like there's something about running this as a Linux container that I'm forgetting in order to make this run properly.
Thank you!
Here is a full example of how to use IHostApplicationLifetime.StopApplication().
void Main()
{
var host = Host.CreateDefaultBuilder()
.ConfigureServices((context, services) =>
{
services.AddHostedService<MyService>();
})
.Build();
Console.WriteLine("Starting service");
host.Run();
Console.WriteLine("Ended service");
}
// You can define other methods, fields, classes and namespaces here
public class MyService : BackgroundService
{
private readonly IHostApplicationLifetime _lifetime;
private readonly Random _rnd = new Random();
public MyService(IHostApplicationLifetime lifetime)
{
_lifetime = lifetime;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
while (true)
{
stoppingToken.ThrowIfCancellationRequested();
var nextNumber = _rnd.Next(10);
if (nextNumber < 8)
{
Console.WriteLine($"We have number {nextNumber}");
}
else
{
throw new Exception("Number too high");
}
await Task.Delay(1000);
}
}
// If the application is shutting down, ignore it
catch (OperationCanceledException e) when (e.CancellationToken == stoppingToken)
{
Console.WriteLine("Application is shutting itself down");
}
// Otherwise, we have a real exception, so must ask the application
// to shut itself down.
catch (Exception e)
{
Console.WriteLine("Oh dear. We have an exception. Let's end the process.");
// Signal to the OS that this was an error condition by
// setting the exit code.
Environment.ExitCode = 1;
_lifetime.StopApplication();
}
}
}
Typical output from this program will look like:
Starting service
We have number 0
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: C:\Users\rowla\AppData\Local\Temp\LINQPad6\_spgznchd\shadow-1
We have number 2
Oh dear. We have an exception. Let's end the process.
info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...
Ended service
Related
I need to run different instances of a Worker Service (BackgroundService): same code but different configuration.
The number of instances and the start and stop of every instance will change during the running process.
So my choice was to write 2 programs:
WorkerProgram: the Worker Service
MainProgram: manage (start and stop) of every different instance of WorkerProgram
Sample code of WorkerProgram:
var host = Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddSingleton(new Instance { Id = int.Parse(args[0])});
services.AddHostedService<Worker>();
})
.Build();
await host.RunAsync();
public class Worker : BackgroundService
{
private readonly Instance _instance;
public Worker(Instance instance)
{
_instance = instance;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
_logger.LogInformation("Running {}", _instance.Id);
await Task.Delay(10000, stoppingToken);
_logger.LogInformation("END");
}
}
}
MainProgram starts every WorkerProgram using
var process = new Process();
process.StartInfo.Filename = "WorkerProgram.exe";
process.StartInfo.Arguments = i.ToString();
process.Start();
so every Worker Service run in a different process.
But when it needs to stop them, the command
process.Kill();
will Kill the process in the middle of run.
How can I stop the process in a better way (eg. using CancellationToken)?
Thank you!
My current (bad) solution is to create a placeholder file for every BackgroundService. And check the existence of the file at every cicle.
untested, but:
cancellationToken.ThrowIfCancellationRequested();
process.Start();
using (cancellationToken.Register(static state => {
try {
(state as Process)?.Close(); // or Kill, or both; your choice
} catch (Exception ex) {
// best efforts only
Debug.WriteLine(ex.Message);
}
}, process))
{
await process.WaitForExitAsync(cancellationToken);
}
The Register here will deal with killing the process in the cancellation case; if the external process exits before then, we will unregister via the using after the WaitForExitAsync, and won't attempt to kill it.
Please help to understand what's going on.
I have async method. It return Task<T>.
I need wait this method result BUT with correct stop by Ctrl+C.
The problem is:
If I wait in this way - all works fine:
var result = **await Task<T>**
If I wait in this way - main process suddenly stoped:
var result = **Task<T>.Result**
To show it in code, I write small test (sorry - I can't make it smaller)
using System;
using System.Threading.Tasks;
namespace StopTest4
{
internal static class ConsoleApp
{
private static readonly TaskCompletionSource<bool> GetFirstString = new TaskCompletionSource<bool>();
private static void OnCancelKeyPressed(object sender, ConsoleCancelEventArgs args)
{
Console.WriteLine("Got Stop Signal (by Crt+C)");
GetFirstString.TrySetResult(false);
args.Cancel = true;
}
public static int Main(string[] args)
{
try
{
var appTask = RunAsync();
Console.CancelKeyPress += OnCancelKeyPressed;
return appTask.Result;
}
catch (Exception e)
{
Console.WriteLine($"Got exception while run main task of the App - {e}");
}
return -2;
}
private static async Task<int> RunAsync()
{
Console.WriteLine("RunAsync START");
try
{
await LoadWorksFine().ConfigureAwait(false);
//await LoadWithProcessSuddenlyClose().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine($"Got Exception - {e}");
}
Console.WriteLine("Stopped!");
return 0;
}
private static Task LoadWithProcessSuddenlyClose()
{
try
{
Console.WriteLine($"Befor LoadString");
var result = LoadString().Result;
Console.WriteLine($"After LoadString - result=\"{result}\"");
}
catch (Exception e)
{
Console.WriteLine($"LOAD: Got Exception - {e}");
}
return Task.CompletedTask;
}
private static async Task LoadWorksFine()
{
try
{
Console.WriteLine($"Befor LoadString");
var result = await LoadString();
Console.WriteLine($"After LoadString - result=\"{result}\"");
}
catch (Exception e)
{
Console.WriteLine($"LOAD: Got Exception - {e}");
}
}
private static async Task<string> LoadString()
{
Console.WriteLine($"LoadString: Start");
try
{
if (!await GetFirstString.Task.ConfigureAwait(false))
{
Console.WriteLine($"LoadString: waited task Failed!");
return "false";
}
Console.WriteLine($"LoadString: waited task Success!");
return "true";
}
catch (Exception e)
{
Console.WriteLine($"LoadString: Got Exception - {e}");
}
return null;
}
}
}
How to check:
If You run the test and press Ctrl+C you will see such output:
% dotnet StopTest4.dll
RunAsync START
Befor LoadString
LoadString: Start
^CGot Stop Signal (by Crt+C) <---- Here press Ctrl+C
LoadString: waited task Failed!
After LoadString - result="false"
Stopped!
%
All works fine!
Now in method RunAsync comment LoadWorksFine() and uncomment LoadWithProcessSuddenlyClose(). Like this:
private static async Task<int> RunAsync()
{
Console.WriteLine("RunAsync START");
try
{
//await LoadWorksFine().ConfigureAwait(false);
await LoadWithProcessSuddenlyClose().ConfigureAwait(false);
}
catch (Exception e)
{
Console.WriteLine($"Got Exception - {e}");
}
Console.WriteLine("Stopped!");
return 0;
}
Now run the test and press Ctrl+C you will see such output:
% dotnet StopTest4.dll
RunAsync START
Befor LoadString
LoadString: Start
^C <---- Here press Ctrl+C
%
There are no any exceptions or errors - just process stopped.
It works on MacOS, Linux and Windows.
Who knows why? As for me - both variants correct.
Thank You!
P.S.: I want second variant - I need to call async method in synchronous method.
I post the question on dotnet git.
And got the answer:
Because your example is synchronously blocking the RunAsync method
waiting for the handler to be invoked, but by that point you haven't
hooked up the handler. So when ctrl-C is pressed, there isn't a
handler to cancel the ctrl-C, and the OS kills the process.
And yes - in this code:
public static int Main(string[] args)
{
try
{
var appTask = RunAsync();
Console.CancelKeyPress += OnCancelKeyPressed;
return appTask.Result;
}
catch (Exception e)
{
Console.WriteLine($"Got exception while run main task of the App - {e}");
}
return -2;
}
line Console.CancelKeyPress += OnCancelKeyPressed; never called.
If put it BEFOR var appTask = RunAsync(); all works fine in both variants.
But now I don't understand WHY in second variant Console.CancelKeyPress += OnCancelKeyPressed; never called.
As I understand, var appTask = RunAsync(); only create Task.
Task waiting here - return appTask.Result;!
WHY var appTask = RunAsync(); synchronously blocked? It is a TASK...
UPDATE
Now I understood!
You call RunAsync:
var appTask = RunAsync();
RunAsync calls LoadWithProcessSuddenlyClose:
await LoadWithProcessSuddenlyClose().ConfigureAwait(false);
Note that this is exactly the same as if you'd written:
Task t = LoadWithProcessSuddenlyClose();
await t.ConfigureAwait(false);
meaning you're calling LoadWithProcessSuddenlyClose synchronously from RunAsync, and there are no awaits that complete asynchronously prior to it, so your Main method has synchronously invoked LoadWithProcessSuddenlyClose. Then LoadWithProcessSuddenlyClose synchronously blocks:
var result = LoadString().Result;
waiting for the ctrl-C handler to be invoked. You've created a deadlock: you're blocking waiting for something that can only happen after this call returns, because it's only hooked up after this call returns, but this call will never return because it requires the hooked up thing to be invoked to do so.
This is the code where I initialize my custom logger, the way it works is I instanciate the class frmLog in the MainForm, and this, which is the frmlog instanciate the logger. The logger is nothing fancy, it works with a queue from the server, which I copy while I lock it, and then i dump the queue into a txt. What failed was the process of the queue, but I dont want to take any chances.
What I expect to happen with this code is: Every time there is a problem in the logger which makes it throw an exception which is not handled, to catch it here and restart it. Would this work as intended?
Server Server;
LogHelper logger;
int counter = 0;
public frmLog(Server server)
{
InitializeComponent();
Server = server;
}
void inicializarlog()
{
logger = new LogHelper(rtbLog, Server, this);
}
private void btnSalirLog_Click(object sender, EventArgs e)
{
this.Hide();
}
private void frmLog_Load(object sender, EventArgs e)
{
MainAsync();
}
private async Task MainAsync()
{
try
{
await Task.Factory.StartNew(() => inicializarlog());
}
catch (Exception ex)
{
Console.WriteLine("Fallo el asincronico: " + ex.Message);
if(counter++ >= 5)
MainAsync();
else
Console.WriteLine("Fatal error, shutting down the logger");
}
}
Edit: The retry and the polly library its if the work fails, since what im doing is creating an instance of a logger, it will always finish, the problem is after the fact. unless i got a problem in the server
I have a headless UWP app that is supposed to get data from a sensor every 10 minutes and send it to the cloud.
When I run the code from the headless app on the Raspberry Pi it stops measuring after 3 or 4 hours, no error (I have a lot of logs). It is exactly 3 or 4 hours. If I start the app at 8, at 11 or 12 it just stops...
It looks like it is stopped because I have cancellation token in place that worked well in tests, but here it is not firing anymore. On the App manager in the Device Portal it appears that the app is running.
I also noticed in the Performance page in the Device Portal that the memory goes down with about 8 MB during the measurements.
The strange thing is that I ran the same code from a headed app on the RPi and on a laptop and it went very well. It worked continuously for over 16 hours until I stopped it. On both the laptop and the RPi there was no memory issue, the app used the same amount of RAM over the whole period.
What could cause this behavior when running as a headless app?
Here is how I call the code from the headless app:
BackgroundTaskDeferral deferral;
private ISettingsReader settings;
private ILogger logger;
private IFlowManager<PalmSenseMeasurement> flow;
private IServiceProvider services;
IBackgroundTaskInstance myTaskInstance;
public async void Run(IBackgroundTaskInstance taskInstance)
{
taskInstance.Canceled += TaskInstance_Canceled;
deferral = taskInstance.GetDeferral();
myTaskInstance = taskInstance;
try
{
SetProperties();
var flowTask = flow.RunFlowAsync();
await flowTask;
}
catch (Exception ex)
{
logger.LogCritical("#####---->Exception occured in StartupTask (Run): {0}", ex.ToString());
}
}
private void SetProperties()
{
services = SensorHubContainer.Services;
settings = services.GetService<ISettingsReader>();
flow = services.GetService<IFlowManager<PalmSenseMeasurement>>();
logger = services.GetService<ILogger<StartupTask>>();
}
private void TaskInstance_Canceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
{
logger.LogDebug("StartupTask.TaskInstance_Canceled() - {0}", reason.ToString());
deferral.Complete();
}
And here is how I call the code from the headed app:
private async Task GetMeasurementsAsync()
{
try
{
flow = services.GetService<IFlowManager<PalmSenseMeasurement>>();
await flow.RunFlowAsync();
}
catch (Exception ex)
{
Measurements.Add(new MeasurementResult() { ErrorMessage = ex.Message });
}
}
The RunFlowAsync method looks like this:
public async Task RunFlowAsync()
{
var loopInterval = settings.NoOfSecondsForLoopInterval;
while (true)
{
try
{
logger.LogInformation("Starting a new loop in {0} seconds...", loopInterval);
//check for previous unsent files
await resender.TryResendMeasuresAsync();
await Task.Delay(TimeSpan.FromSeconds(loopInterval));
await DoMeasureAndSend();
logger.LogInformation("Loop finished");
}
catch (Exception ex)
{
logger.LogError("Error in Flow<{0}>! Error {1}", typeof(T).FullName, ex);
#if DEBUG
Debug.WriteLine(ex.ToString());
#endif
}
}
}
The problem was from a 3rd party library that I had to use and it had to be called differently from a headless app.
Internally it was creating its own TaskScheduler if SynchronizationContext.Current was null.
I am using Quartz.Net to implement some asynchronous processing within the IIS worker process (IIS 8.5). One particular job may take more 10 minutes to run and does a lot of processing.
The following pieces of code illustrate how I am handling job life cycle.
Schedule definition
public class JobScheduler
{
private static IScheduler _quartzScheduler;
public static void Start()
{
_quartzScheduler = new StdSchedulerFactory().GetScheduler();
_quartzScheduler.JobFactory = new NinjectJobFactory();
ScheduleTheJob(_quartzScheduler);
_quartzScheduler.Context.Add("key", "scheduler");
_quartzScheduler.Start();
}
private static void ScheduleTheJob(IScheduler scheduler)
{
IJobDetail job = JobBuilder.Create<JobClass>().UsingJobData("JobKey", "JobValue").Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("JobTrigger", "JobGroup").UsingJobData("JobKey", "JobTrigger").StartNow()
.WithSimpleSchedule(x => x
// 30 minutes for refresh period
.WithIntervalInSeconds(ConfigService.JobRefreshPeriod)
.RepeatForever())
.Build();
scheduler.ScheduleJob(job, trigger);
}
public static void StopScheduler(bool waitForCompletion)
{
_quartzScheduler.Shutdown(waitForCompletion);
}
}
Application pool shutdown handling
public class ApplicationPoolService : IApplicationPoolService
{
public bool IsShuttingDown()
{
return System.Web.Hosting.HostingEnvironment.ShutdownReason != ApplicationShutdownReason.None;
}
public ApplicationShutdownReason GetShutdownReason()
{
return System.Web.Hosting.HostingEnvironment.ShutdownReason;
}
}
public class HostingEnvironmentRegisteredObject : IRegisteredObject
{
public void Stop(bool immediate)
{
if (immediate)
return;
JobScheduler.StopScheduler(waitForCompletion: true);
var logger = NinjectWebCommon.Kernel.Get<ILoggingService>();
var appPoolService = NinjectWebCommon.Kernel.Get<IApplicationPoolService>();
var reason = appPoolService.GetShutdownReason().ToString();
logger.Log(LogLevel.Info, $"HostingEnvironmentRegisteredObject.stop called with shutdown reason {reason}");
}
}
Global.asax.cs wiring up
protected void Application_Start()
{
JobScheduler.Start();
HostingEnvironment.RegisterObject(new HostingEnvironmentRegisteredObject());
}
protected void Application_Error()
{
Exception exception = Server.GetLastError();
Logger.Log(LogLevel.Fatal, exception, "Application global error");
}
protected void Application_End(object sender, EventArgs e)
{
// stopping is now triggered in HostingEnvironmentRegisteredObject
// JobScheduler.StopScheduler(false);
// get shutdown reason
var appPoolService = NinjectWebCommon.Kernel.Get<IApplicationPoolService>();
var reason = appPoolService.GetShutdownReason().ToString();
Logger.Log(LogLevel.Info, $"Application_End called with shutdown reason {reason}");
}
Job step description
if (ApplicationPoolService.IsShuttingDown())
{
Logger.Log(LogLevel.Info, "(RefreshEnvironmentImportingSystemData) Application pool is shutting down");
return;
}
// about 20-30 steps may be here
environments.ForEach(env =>
{
if (ApplicationPoolService.IsShuttingDown())
return;
// do heavy processing for about 2 minutes (worst case), typically some 10-20s
}
// one job step may allocate several hundreds of MB, so GC is called to reclaim some memory sooner
// it takes a few seconds (worst case)
GC.Collect();
GC.WaitForPendingFinalizers();
GC.WaitForFullGCComplete();
GC.Collect();
Initially, I was stopping the scheduler when Application_End was called, but I realized that was called when application pool was about to be killed, so I move it when application pool was notified that its shutdown has been started.
I have left application pool with its default value for Shutdown time limit (90 seconds).
Job is configured to not allow concurrent executions.
I want to achieve the following:
avoid forced killing of job during actual execution
minimize the time when two worker processes run in the same time (shutting down one in parallel with the one just started to handle new requests)
Question: did I manage correctly the scheduled jobs or can I make improvements?