We Keep Coding

High load ASP.NET application running as Windows Service not stopping

An asp.net application (running on Kestrel) running as a Windows Service has trouble stopping during high load.
The setup:
[...]
var config = WebHostBuilderHelper.GetConfig();
var hostBuilder = new WebHostBuilder().UseKestrel(options => options.ConfigureEndpoints())
.UseConfiguration(config)
.UseStartup<Startup>()
.UseSerilog((hostingContext, loggerConfiguration) => loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration))
.UseContentRoot(AppDomain.CurrentDomain.BaseDirectory);
if (Debugger.IsAttached || args.ToList()
.Contains("console"))
{
var host = hostBuilder.Build();
host.Run();
}
else
{
var host = hostBuilder.UseContentRoot(pathToContentRoot)
.Build();
host.RunAsCustomService();
}
RunAsCustomService:
public static class WebHostServiceExtensions
{
public static void RunAsCustomService(this IWebHost host)
{
var webHostService = new CustomWebHostService(host);
ServiceBase.Run(webHostService);
}
}
internal class CustomWebHostService : WebHostService
{
private ILogger _logger;
public CustomWebHostService(IWebHost host) : base(host)
{
Console.Write("CustomWebHostService start");
_logger = host.Services
.GetRequiredService<ILogger<CustomWebHostService>>();
}
[...]
}
When stopping the service, it correctly receives the Stop-signal from Windows (can be verified by both logs and attaching debugger) -- and it correctly stops very quickly when not under load.
During OnStop(); StopAsync() is called on the host, and then, eventually, it is disposed. I'm guessing the StopAsync does not stop very well when there are a large amount of requests being made to at the same time. Some signal being lost perhaps?
During normal operation all objects are correctly cleaned up or otherwise disposed.
Best regards,

How to implement .net core IHostedService that runs one time only

I know a IHostedService that runs only one time sounds like a console application, but the reason I want to use it instead of a plain console application is:
.net core introduces Generic Host for running non-http application
A plain console application does not have DI, Logger, Configurations ready to use
By using the following code, I'm able to somewhat achieve this one-time behaviour, however, I could not find a way to gracefully exit the app after it finishes.
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) => { services.AddHostedService<StartUp>(); });
}
Where StartUp is a simple IHostedService
public class StartUp:IHostedService
{
private ILogger<StartUp> _logger;
public StartUp(ILogger<StartUp> logger)
{
_logger = logger;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("start async");
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("stop async");
return Task.CompletedTask;
}
}
How can I stop the app gracefully?
Or if this is completely wrong, how should I implement this one-time application?

Yes, you can achieve that by injecting IHostApplicationLifetime into your hosted service.
Example:
public class StartUp:IHostedService
{
private readonly IHostApplicationLifetime _host;
private ILogger<StartUp> _logger;
public StartUp(IHostApplicationLifetime host, ILogger<StartUp> logger)
{
_host = host;
_logger = logger;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("start async");
_host.StopApplication();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("stop async");
return Task.CompletedTask;
}
}
You can also set exit code by setting Environment.ExitCode.For example:
Environment.ExitCode = 0;
_host.StopApplication();

If you are just using IHostedService as a workaround for the missing DI and ILogger you can also setup DI with the ILogger and IConfiguration directly without IHostedService
public class Program
{
public static async Task Main(string[] args)
{
var configBuilder = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", optional: true);
var config = configBuilder.Build();
var sp = new ServiceCollection()
.AddLogging(b => b.AddConsole())
.AddSingleton<IConfiguration>(config)
.AddSingleton<IFooService, FooService>()
.BuildServiceProvider();
var logger = sp.GetService<ILoggerFactory>().CreateLogger<Program>();
logger.LogDebug("Starting");
var bar = sp.GetService<IFooService>();
await bar.DoAsync();
}
}
With this setup your code is just running once, can resolve every service you register in the ServiceCollection and no need of a Host to start
Example: .netfiddle

You have to call IHostApplicationLifetime.StopApplication() for the application to close. You can inject IHostApplicationLifetime and call StopApplication() when your done.

Another implementation might be using IHostBuilder, but without calling run method.
static Task Main(string[] args)
{
using IHost host = CreateHostBuilder(args).Build();
var manager = host.Services.GetRequiredService<IManager>();
manager.DoSomething();
// we don't have registered hosted services therefore we don't call host.RunAsync();
return Task.CompletedTask; // or change signature to void and remove return
}
static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((_, services) =>
services
.AddScoped<IFoo, Foo>()
.AddScoped<IBar, Bar>()
.AddSingleton<IManager, Manager>());

In a .net core 3.0 BackgroundService app, why is my configuration object empty when running as service, but not as console app?

I have a .net Core 3.0 BackgroundService application that works fine when running in console mode, but once i deploy as a service the configuration object that should be loaded from appsettings.json is empty. What gives?
Program.cs
public class Program
{
public static async System.Threading.Tasks.Task Main(string[] args)
{
var hostbuilder = new HostBuilder()
.UseWindowsService()
.ConfigureAppConfiguration((hostingContext, config) =>
{
config
.SetBasePath(Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location))
.AddJsonFile("appsettings.json");
})
.ConfigureLogging(
options => options.AddFilter<EventLogLoggerProvider>(level => level >= LogLevel.Information))
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Importer>().Configure<EventLogSettings>(config =>
{
config.LogName = "Application";
config.SourceName = "Importer";
});
});
#if (DEBUG)
await hostbuilder.RunConsoleAsync();
#else
await hostbuilder.RunAsServiceAsync();
#endif
}
}
Extension Method for IhostBuilder to run service
public static class ServiceBaseLifetimeHostExtensions
{
public static IHostBuilder UseServiceBaseLifetime(this IHostBuilder hostBuilder)
{
return hostBuilder.ConfigureServices((hostContext, services) => services.AddSingleton<IHostLifetime, ServiceBaseLifetime>());
}
public static Task RunAsServiceAsync(this IHostBuilder hostBuilder, CancellationToken cancellationToken = default)
{
return hostBuilder.UseServiceBaseLifetime().Build().RunAsync(cancellationToken);
}
}
ServiceBaseLifetime class to handle service lifecycle
public class ServiceBaseLifetime : ServiceBase, IHostLifetime
{
private readonly TaskCompletionSource<object> _delayStart = new TaskCompletionSource<object>();
public ServiceBaseLifetime(IHostApplicationLifetime applicationLifetime)
{
ApplicationLifetime = applicationLifetime ?? throw new ArgumentNullException(nameof(applicationLifetime));
}
private IHostApplicationLifetime ApplicationLifetime { get; }
public Task WaitForStartAsync(CancellationToken cancellationToken)
{
cancellationToken.Register(() => _delayStart.TrySetCanceled());
ApplicationLifetime.ApplicationStopping.Register(Stop);
new Thread(Run).Start(); // Otherwise this would block and prevent IHost.StartAsync from finishing.
return _delayStart.Task;
}
private void Run()
{
try
{
Run(this); // This blocks until the service is stopped.
_delayStart.TrySetException(new InvalidOperationException("Stopped without starting"));
}
catch (Exception ex)
{
_delayStart.TrySetException(ex);
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
Stop();
return Task.CompletedTask;
}
// Called by base.Run when the service is ready to start.
protected override void OnStart(string[] args)
{
_delayStart.TrySetResult(null);
base.OnStart(args);
}
// Called by base.Stop. This may be called multiple times by service Stop, ApplicationStopping, and StopAsync.
// That's OK because StopApplication uses a CancellationTokenSource and prevents any recursion.
protected override void OnStop()
{
ApplicationLifetime.StopApplication();
base.OnStop();
}
}
The actual implementation of the service is irrelevant other than the constructor, which takes the logger and configuration through DI.
private readonly ILogger<Importer> _logger;
private readonly IConfiguration _configuration;
public Importer(IConfiguration configuration, ILogger<Importer> logger)
{
_logger = logger;
_configuration = configuration;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation($"Why is {_configuration["Key1"]} empty?");
}
appsettings.json
{
"Key1":"some value"
}
When i run through debug the console app starts up and runs and logs and has the configuration loaded from appsettings. When i deploy as a service the configuration object is empty.
Notes: The appsettings file is being read, i can tell this by changing the name of it and it throws an exception for file not found. The appsettings file is also not empty.

My issue appears to be some kind of async race condition problem (I am guessing, not positive). The first tick through ExecuteAsync the configuration is not loaded, but the second time through it is. I had the service dying if it encountered that exception, so i never got it to tick a second time.

This appears to be an XY problem and is worthy of a refactor
Create a strongly typed model to hold the desired settings
public class ImporterSettings {
public string Key1 { get; set; }
}
Refactor hosted service to depend on the settings since tightly coupling services to IConfiguration is a code smell in my opinion
private readonly ILogger<Importer> _logger;
private readonly ImporterSettnigs settings;
public Importer(ImporterSettnigs settings, ILogger<Importer> logger) {
_logger = logger;
this.settings = settings;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
_logger.LogInformation($"This is Key1: {settings.Key1}");
}
Now properly configure start up to use the provided configuration
public class Program {
public static async System.Threading.Tasks.Task Main(string[] args) {
var hostbuilder = new HostBuilder()
.UseWindowsService()
.ConfigureAppConfiguration((hostingContext, config) => {
var path = Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location);
config
.SetBasePath(path)
.AddJsonFile("appsettings.json");
})
.ConfigureLogging(
options => options.AddFilter<EventLogLoggerProvider>(level =>
level >= LogLevel.Information)
)
.ConfigureServices((hostContext, services) => {
//get settings from app configuration.
ImporterSettings settings = hostContext.Configuration.Get<ImporterSettings>();
services
.AddSingleton(settings) //add to service collection
.AddHostedService<Importer>()
.Configure<EventLogSettings>(config => {
config.LogName = "Application";
config.SourceName = "Importer";
});
});
#if (DEBUG)
await hostbuilder.RunConsoleAsync();
#else
await hostbuilder..Build().RunAsync();
#endif
}
}

Performing a health check in .NET Core Worker Service

How can I implement health checks in a .NET Core Worker Service?
The service will be run inside Docker and needs to be able to check the health of the service.

Another way of doing this is to implement IHealthCheckPublisher.
The benefits of this approach is the ability to re-use your existing IHealthChecks or integration with 3rd party libraries that rely on IHealthCheck interface (like this one).
Though you still target Microsoft.NET.Sdk.Web as the SDK you don't need to add any asp.net specifics.
Here is an example:
public static IHostBuilder CreateHostBuilder(string[] args)
{
return Host
.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services
.AddHealthChecks()
.AddCheck<RedisHealthCheck>("redis_health_check")
.AddCheck<RfaHealthCheck>("rfa_health_check");
services.AddSingleton<IHealthCheckPublisher, HealthCheckPublisher>();
services.Configure<HealthCheckPublisherOptions>(options =>
{
options.Delay = TimeSpan.FromSeconds(5);
options.Period = TimeSpan.FromSeconds(5);
});
});
}
public class HealthCheckPublisher : IHealthCheckPublisher
{
private readonly string _fileName;
private HealthStatus _prevStatus = HealthStatus.Unhealthy;
public HealthCheckPublisher()
{
_fileName = Environment.GetEnvironmentVariable(EnvVariableNames.DOCKER_HEALTHCHECK_FILEPATH) ??
Path.GetTempFileName();
}
public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
{
// AWS will check if the file exists inside of the container with the command
// test -f $DOCKER_HEALTH_CHECK_FILEPATH
var fileExists = _prevStatus == HealthStatus.Healthy;
if (report.Status == HealthStatus.Healthy)
{
if (!fileExists)
{
using var _ = File.Create(_fileName);
}
}
else if (fileExists)
{
File.Delete(_fileName);
}
_prevStatus = report.Status;
return Task.CompletedTask;
}
}

I don't think is worth it to change SDK to Microsoft.NET.Sdk.Web. You will include additional middlewares just because of one health check? No thanks ...
What you could do is to use a different protocol like TCP.
The general idea is:
Create a separate background service that creates a TCP server (take a look at TcpListener.cs)
When you receive a request you have two options: if the application is healthy accept TCP connection otherwise reject it.
If you use containers your orchestrator should have an option to call it over TCP (in k8s there is a property tcpSocket)
If you need more detailed information you may check: Monitoring Health of ASP.NET Core Background Services With TCP Probes on Kubernetes
Cheers!

I think that you should also consider to retain the Microsoft.NET.Sdk.Worker.
Don't change the whole sdk just because of the health checks.
Then you can create a backgroundservice (just like the main worker), in order to update a file to write for example the current timestamp. An example of the background health check worker would be:
public class HealthCheckWorker : BackgroundService
{
private readonly int _intervalSec;
private readonly string _healthCheckFileName;
public HealthCheckWorker(string healthCheckFileName, int intervalSec)
{
this._intervalSec = intervalSec;
this._healthCheckFileName = healthCheckFileName;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (true)
{
File.WriteAllText(this._healthCheckFileName, DateTime.UtcNow.ToString());
await Task.Delay(this._intervalSec * 1000, stoppingToken);
}
}
}
Then you can add a extension method like this:
public static class HealthCheckWorkerExtensions
{
public static void AddHealthCheck(this IServiceCollection services,
string healthCheckFileName, int intervalSec)
{
services.AddHostedService<HealthCheckWorker>(x => new HealthCheckWorker(healthCheckFileName, intervalSec));
}
}
With this you can add in services the health check support
.ConfigureServices(services =>
{
services.AddHealthCheck("hc.txt", 5);
})

Add HTTPListener and expose the health checks endpoints.
Using HTTPListener does not require adding Microsoft.NET.Sdk.Web SDK.
Program.cs
using Consumer;
IHost host = Host.CreateDefaultBuilder(args)
.ConfigureServices(services =>
{
services.AddHostedService<Worker>();
services.AddHostedService<HttpHealthcheck>();
})
.Build();
await host.RunAsync();
HttpHealthcheck.cs
using System.Net;
using System.Text;
namespace Consumer;
public class HttpHealthcheck : BackgroundService
{
private readonly ILogger<Worker> _logger;
private readonly HttpListener _httpListener;
private readonly IConfiguration _configuration;
public HealthcheckHttpListener(ILogger<Worker> logger, IConfiguration configuration)
{
_logger = logger;
_configuration = configuration;
_httpListener = new HttpListener();
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_httpListener.Prefixes.Add($"http://*:5001/healthz/live/");
_httpListener.Prefixes.Add($"http://*:5001/healthz/ready/");
_httpListener.Start();
_logger.LogInformation($"Healthcheck listening...");
while (!stoppingToken.IsCancellationRequested)
{
HttpListenerContext ctx = null;
try
{
ctx = await _httpListener.GetContextAsync();
}
catch (HttpListenerException ex)
{
if (ex.ErrorCode == 995) return;
}
if (ctx == null) continue;
var response = ctx.Response;
response.ContentType = "text/plain";
response.Headers.Add(HttpResponseHeader.CacheControl, "no-store, no-cache");
response.StatusCode = (int)HttpStatusCode.OK;
var messageBytes = Encoding.UTF8.GetBytes("Healthy");
response.ContentLength64 = messageBytes.Length;
await response.OutputStream.WriteAsync(messageBytes, 0, messageBytes.Length);
response.OutputStream.Close();
response.Close();
}
}
}

What I've done to accomplish this is add Microsoft.NET.Sdk.Web to my Worker, and then configured a web host to run alongside the worker:
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(builder =>
{
builder.UseStartup<Startup>();
})
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Worker>();
services.AddLogging(builder =>
builder
.AddDebug()
.AddConsole()
);
});
With that done, all that's left to do is map the health check endpoint as you normally would with ASP.NET Core.

Task.Delay() triggers cancellation of application lifetime

I have faced a situation when Task.Delay() method would trigger an event of cancellation in IApplicationLifetime. Here is the code:
static async Task Main(string[] args)
{
Console.WriteLine("Starting");
await BuildWebHost(args)
.RunAsync();
Console.WriteLine("Press any key to exit..");
Console.ReadKey();
}
private static IHost BuildWebHost(string[] args)
{
var hostBuilder = new HostBuilder()
.ConfigureHostConfiguration(config =>
{
config.AddEnvironmentVariables();
config.AddCommandLine(args);
})
.ConfigureAppConfiguration((hostContext, configApp) =>
{
configApp.SetBasePath(Directory.GetCurrentDirectory());
configApp.AddCommandLine(args);
})
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<BrowserWorkerHostedService>();
services.AddHostedService<EmailWorkerHostedService>();
})
.UseConsoleLifetime();
return hostBuilder.Build();
}
and here are hosted services which are stopping abnormally:
public class BrowserWorkerHostedService : BackgroundService
{
private IApplicationLifetime _lifetime;
private IHost _host;
public BrowserWorkerHostedService(
IApplicationLifetime lifetime,
IHost host)
{
this._lifetime = lifetime;
this._host = host;
}
protected override async Task ExecuteAsync(CancellationToken stopToken)
{
while (!_lifetime.ApplicationStarted.IsCancellationRequested
&& !_lifetime.ApplicationStopping.IsCancellationRequested
&& !stopToken.IsCancellationRequested)
{
Console.WriteLine($"{nameof(BrowserWorkerHostedService)} is working. {DateTime.Now.ToString()}");
//lifetime.StopApplication();
//await StopAsync(stopToken);
await Task.Delay(1_000, stopToken);
}
Console.WriteLine($"End {nameof(BrowserWorkerHostedService)}");
await _host.StopAsync(stopToken);
}
}
public class EmailWorkerHostedService : BackgroundService
{
private IApplicationLifetime _lifetime;
private IHost _host = null;
public EmailWorkerHostedService(
IApplicationLifetime lifetime,
IHost host)
{
this._lifetime = lifetime;
this._host = host;
}
protected override async Task ExecuteAsync(CancellationToken stopToken)
{
while (!_lifetime.ApplicationStarted.IsCancellationRequested
&& !_lifetime.ApplicationStopping.IsCancellationRequested
&& !stopToken.IsCancellationRequested)
{
Console.WriteLine($"{nameof(EmailWorkerHostedService)} is working. {DateTime.Now.ToString()}");
await Task.Delay(1_000, stopToken);
}
Console.WriteLine($"End {nameof(EmailWorkerHostedService)}");
await _host.StopAsync(stopToken);
}
}
I would like my services to be running, unless lifetime.StopApplication() is triggered. However, hosted services are stopped, because lifetime.ApplicationStarted.IsCancellationRequested variable is set to true upon a second itteration.. Even though, in theory, I have no code that explicitly aborts the application.
Log will look like this:
Starting BrowserWorkerHostedService is working. 09.07.2019 17:03:53
EmailWorkerHostedService is working. 09.07.2019 17:03:53
Application started. Press Ctrl+C to shut down.
Hosting environment: Production
Content root path: xxxx
End EmailWorkerHostedService
End BrowserWorkerHostedService
Is there a good explanation why Task.Delay() triggers ApplicationStarted cancellation event?

You are misusing IApplicationLifetime events. Their purpose is to give you the ability to associate some action with them. For example you want to start message queue listening only when you application is fully started. You are going to do it like this:
_applicationLifetime.ApplicationStarted. Register(StartListenMq);
I think using CancellationTokens here wasn't the best idea, but it the way it was implemented.
When you want to cancel your HostedService you should check only token received in ExecuteAsync method. The flow will look look this:
IApplicationLifetime.StopApplication() => will trigger IApplicationLifetime.ApplicationStopping => will trigger IHostedService.StopAsync() => will stopToken
And now for your question: why it happens on await Task.Delay()? Look again at BackgroundService.StartAsync()
public virtual Task StartAsync(CancellationToken cancellationToken)
{
// Store the task we're executing
_executingTask = ExecuteAsync(_stoppingCts.Token);
// If the task is completed then return it, this will bubble cancellation and failure to the caller
if (_executingTask.IsCompleted)
{
return _executingTask;
}
// Otherwise it's running
return Task.CompletedTask;
}
This code does't await ExecuteAsync. At the moment you call async operation in your code StartAsync() will continue to run. Somewhere in it's callstack it will trigger ApplicationStarted and since you are listening to it you will get _lifetime.ApplicationStarted.IsCancellationRequested = true.