High load ASP.NET application running as Windows Service not stopping - c#

An asp.net application (running on Kestrel) running as a Windows Service has trouble stopping during high load.
The setup:
[...]
var config = WebHostBuilderHelper.GetConfig();
var hostBuilder = new WebHostBuilder().UseKestrel(options => options.ConfigureEndpoints())
.UseConfiguration(config)
.UseStartup<Startup>()
.UseSerilog((hostingContext, loggerConfiguration) => loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration))
.UseContentRoot(AppDomain.CurrentDomain.BaseDirectory);
if (Debugger.IsAttached || args.ToList()
.Contains("console"))
{
var host = hostBuilder.Build();
host.Run();
}
else
{
var host = hostBuilder.UseContentRoot(pathToContentRoot)
.Build();
host.RunAsCustomService();
}
RunAsCustomService:
public static class WebHostServiceExtensions
{
public static void RunAsCustomService(this IWebHost host)
{
var webHostService = new CustomWebHostService(host);
ServiceBase.Run(webHostService);
}
}
internal class CustomWebHostService : WebHostService
{
private ILogger _logger;
public CustomWebHostService(IWebHost host) : base(host)
{
Console.Write("CustomWebHostService start");
_logger = host.Services
.GetRequiredService<ILogger<CustomWebHostService>>();
}
[...]
}
When stopping the service, it correctly receives the Stop-signal from Windows (can be verified by both logs and attaching debugger) -- and it correctly stops very quickly when not under load.
During OnStop(); StopAsync() is called on the host, and then, eventually, it is disposed. I'm guessing the StopAsync does not stop very well when there are a large amount of requests being made to at the same time. Some signal being lost perhaps?
During normal operation all objects are correctly cleaned up or otherwise disposed.
Best regards,

Related

IHost not returning when Task completed

I'm writing a Windows Service (using .NET Core 3.1), using a BackgroundService and I seem to have an issue when I want to programmatically stop the service.
My main function is as follows
static async Task Main(string[] args)
{
IHost host = null;
try
{
host = CreateService(args).Build();
await host.RunAsync();
Console.WriteLine("Ending");
}
catch (Exception ex)
{
Console.WriteLine("Exception");
}
finally
{
if (host is IAsyncDisposable d) await d.DisposeAsync();
}
}
public static IHostBuilder CreateService(string[] args) =>
Host.CreateDefaultBuilder(args)
.UseWindowsService(options =>
{
options.ServiceName = "My Service";
})
.ConfigureServices((hostContext, services) =>
{
IConfiguration configuration = hostContext.Configuration;
ServiceOptions options = configuration.GetSection("ServiceOptions").Get<ServiceOptions>();
WorkerService sService = new WorkerService();
services.AddSingleton(options);
services.AddSingleton(sService);
services.AddHostedService<WindowsBackgroundService>(service => new WindowsBackgroundService(
service.GetService<WorkerService>(),
service.GetService<ILogger<WindowsBackgroundService>>(),
service.GetService<ServerOptions>()
));
});
The background service is as follows:
public sealed class WindowsBackgroundService : BackgroundService
{
private WorkerService _workerService;
private ServerOptions _options;
private ILogger<WindowsBackgroundService> _logger;
public WindowsBackgroundService(
WorkerService workerService, ILogger<WindowsBackgroundService> logger,
ServiceOptions serviceOptions) => (_workerService, _logger, _options) = (workerService, logger, serviceOptions);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
bool allDone = false;
while (!stoppingToken.IsCancellationRequested && !allDone)
{
await Task.Delay(TimeSpan.FromSeconds(15), stoppingToken);
// Log to watchdog
// Check if we are in the run window
if (_options.InActivePeriod())
{
Console.WriteLine("Process Queue");
var processResult = _workerService.Process(_options);
if (!processResult && _workerService.FatalError)
{
_workerService = null;
allDone = true;
Console.WriteLine("Service Quitting");
}
}
}
Console.WriteLine($"Task Ending {stoppingToken.IsCancellationRequested}");
return;
}
}
}
So everything runs as it should and I run this under the debugger or from the command line (I haven't actually installed it as a Windows Service yet as I'm still writing the code).
The function Process executes correctly. If it encounters an error that cannot be recovered, it sets it's FatalError property that is supposed to be a signal that the whole service should be stopped. The Task does indeed complete (the correct lines are written to console) but the line Console.WriteLine("Ending"); is never executed. It looks like the host.RunAsync(); never returns (unless I do a CTRL+C).
I'm not entirely certain what I am doing wrong at this point. Can anyone point me in the write direction?
Based on the shown code I see nothing that would cause the host to stop.
The hosted service, once started has no bearing on the application host itself. So even when ExecuteAsync is completed, the completion of that function does not mean that the host will stop running.
You could consider injecting IHostApplicationLifetime into the hosted service and explicitly telling the application to stop programmatically;
For example
public sealed class WindowsBackgroundService : BackgroundService {
private WorkerService _workerService;
private ServerOptions _options;
private ILogger<WindowsBackgroundService> _logger;
private IHostApplicationLifetime lifetime;
public WindowsBackgroundService(
WorkerService workerService, ILogger<WindowsBackgroundService> logger,
ServiceOptions serviceOptions, IHostApplicationLifetime lifetime)
=> (_workerService, _logger, _options, this.lifetime) = (workerService, logger, serviceOptions, lifetime);
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
bool allDone = false;
while (!stoppingToken.IsCancellationRequested && !allDone) {
await Task.Delay(TimeSpan.FromSeconds(15), stoppingToken);
// Log to watchdog
// Check if we are in the run window
if (_options.InActivePeriod()) {
Console.WriteLine("Process Queue");
var processResult = _workerService.Process(_options);
if (!processResult && _workerService.FatalError) {
_workerService = null;
allDone = true;
Console.WriteLine("Service Quitting");
lifetime.StopApplication(); //SIGNAL APPLICATION TO STOP
}
}
}
Console.WriteLine($"Task Ending {stoppingToken.IsCancellationRequested}");
return;
}
}
I am also curious about how you configured your services. Why use the factory delegate when everything you resolved manually would have been injected automatically if the default registration was done?
//...
.ConfigureServices((hostContext, services) => {
IConfiguration configuration = hostContext.Configuration;
ServiceOptions options = configuration.GetSection("ServiceOptions").Get<ServiceOptions>();
services.AddSingleton(options);
services.AddSingleton<WorkerService>();
services.AddHostedService<WindowsBackgroundService>();
});
The logging and and host lifetime are added for you by the Host.CreateDefaultBuilder

Stop console application that is using DI and a hosted service for user code

I am building a console application that is supposed to stop running after several hours. I used dependency injection to get instances of my services, and my main service is registered as a Hosted service to act as an entrypoint for my code.
After the application has done it's job and the executable should stop running. If I call StopAsync() on the service, it runs the method but doesn't kill the executable. How do I exit the program?
Program.cs
class Program
{
private static readonly Logger _logger = LogManager.GetCurrentClassLogger();
public static void Main(string[] args)
{
_logger.Info($"starting execution of Program.Main()");
CreateHostBuilder(args).Build().Run();
_logger.Info($"finished execution of Program.Main()");
}
public static IHostBuilder CreateHostBuilder(string[] args)
{
IConfiguration Configurations = new ConfigurationBuilder()
.SetBasePath(Directory.GetParent(AppContext.BaseDirectory).FullName)
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
#if DEBUG
.AddUserSecrets("secrets guid")
#endif
.Build();
return Host.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((hostContext, builder) => builder.AddConfiguration(Configurations))
.ConfigureServices(services =>
{
services.AddHostedService<MainService>();
services.AddScoped<Service1>();
services.AddScoped<Service2>();
services.AddScoped<Service3>();
services.AddDbContext<pig_dbContext>(options =>
options.UseMySql(Configurations.GetConnectionString("XXXX"), builder =>
builder.EnableRetryOnFailure(5, TimeSpan.FromSeconds(10), null)));
});
}
}
MainService.cs
public class MainService: IHostedService
{
private static readonly Logger _logger = LogManager.GetCurrentClassLogger();
public MainService(Service1 service1, Service2 service2, Service3 service3, IServiceProvider services, IConfiguration config)
{
//...
}
public Task StartAsync(CancellationToken cancellationToken)
{
//This method is called by .NET when calling Host.Run()
///Do work...
//This call reaches the method, but does not end execution of the application. I want the application to completely exit.
StopAsync(new CancellationToken());
}
I figured it out: The host kept blocking the main thread, so using
CreateHostBuilder(args).Build().RunAsync();
fixed my issue :)

How to run .NET Core Console app using generic host builder

I am trying to figure out how to use hostbuilder pattern to run a console app (not a windows service). Intent is to keep the flow as similar to a WebApi to keep development practices similar. I have seen samples for using HostedService or BackGroundService, where they want to run it as a windows service. But If I am looking to run a simple console app, where do I specify my entrypoint class and method? From hostbuilder.Build(), I can see Run() and RunAsync() methods. But I am unable to figure out what will it execute?
I have seen other examples of where you can create servicecollection and then use serviceprovider.GetService().SomeMethod() to start the process. But that kind of deviates from what we want to do. So please suggest how to specify startup process. We are using 3.1 .Net Core.
class Program
{
static async void Main(string[] args)
{
var host = CreateHostBuilder(args).Build();
await host.RunAsync();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostBuilderContext, serviceCollection) => new Startup(hostBuilderContext.Configuration).ConfigureServices(serviceCollection))
.UseSerilog()
;
}
EDIT: An update for .NET 6 is below ↓
Not much has changed with .NET 7.
I'd start off with the default worker template. It comes with necessary packages pre-installed. If you already have a project, install Microsoft.Extensions.Hosting package.
dotnet new worker -n MyCli
Then open up the Program.cs and build the host. Remove the Worker hosted service if you don't want to go with the hosted service route.
public class Program
{
public static void Main(string[] args)
{
var host = CreateHostBuilder(args).Build();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
// remove the hosted service
// services.AddHostedService<Worker>();
// register your services here.
});
}
Build your logic:
internal class MyService
{
// you can also inject other services
private ILogger<MyService> _logger;
public MyService(ILogger<MyService> logger)
{
_logger = logger;
}
public void DoSomething()
{
_logger.LogInformation("Doing something");
}
}
Then register the class inside .ConfigureServices method
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddTransient<MyService>();
});
Now you can resolve and call it inside the Main method:
public static void Main(string[] args)
{
var host = CreateHostBuilder(args).Build();
var myService = host.Services.GetRequiredService<MyService>();
myService.DoSomething();
}
.NET 6 update
With .NET 6, boilerplate is reduced significantly. We can rewrite our Program.cs as:
var host = Host.CreateDefaultBuilder(args)
.ConfigureServices(services => { services.AddTransient<MyService>(); })
.Build();
var my = host.Services.GetRequiredService<MyService>();
await my.ExecuteAsync();
class MyService
{
private readonly ILogger<MyService> _logger;
public MyService(ILogger<MyService> logger)
{
_logger = logger;
}
public async Task ExecuteAsync(CancellationToken stoppingToken = default)
{
_logger.LogInformation("Doing something");
}
}
Basically:
Instantiate your host builder and configure your services and whatnot.
Make a class with a method for your programme and register that class as a service.
Build the host, create a scope, get an instance of your class, call your method.
My programme is the mehod MainAsync in my class ProgramAsync.
class Program
{
static void Main(string[] args)
{
using (IHost host = Host.CreateDefaultBuilder(args)
.ConfigureAppConfiguration(cfg =>
{
cfg.AddJsonFile("appsettings.json");
})
.ConfigureServices((context, services) =>
{
services.AddDbContext<EquitiesDbContext>(options => { options.UseSqlServer(context.Configuration.GetConnectionString("Equities")); });
services.AddScoped<ProgramAsync>();
})
.ConfigureLogging((context, cfg) =>
{
cfg.ClearProviders();
cfg.AddConfiguration(context.Configuration.GetSection("Logging"));
cfg.AddConsole();
})
.Build()
)
{
using( IServiceScope scope = host.Services.CreateScope() )
{
ProgramAsync p = scope.ServiceProvider.GetRequiredService<ProgramAsync>();
p.MainAsync().Wait();
}
}
Console.WriteLine("Done.");
}
}

In a .net core 3.0 BackgroundService app, why is my configuration object empty when running as service, but not as console app?

I have a .net Core 3.0 BackgroundService application that works fine when running in console mode, but once i deploy as a service the configuration object that should be loaded from appsettings.json is empty. What gives?
Program.cs
public class Program
{
public static async System.Threading.Tasks.Task Main(string[] args)
{
var hostbuilder = new HostBuilder()
.UseWindowsService()
.ConfigureAppConfiguration((hostingContext, config) =>
{
config
.SetBasePath(Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location))
.AddJsonFile("appsettings.json");
})
.ConfigureLogging(
options => options.AddFilter<EventLogLoggerProvider>(level => level >= LogLevel.Information))
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Importer>().Configure<EventLogSettings>(config =>
{
config.LogName = "Application";
config.SourceName = "Importer";
});
});
#if (DEBUG)
await hostbuilder.RunConsoleAsync();
#else
await hostbuilder.RunAsServiceAsync();
#endif
}
}
Extension Method for IhostBuilder to run service
public static class ServiceBaseLifetimeHostExtensions
{
public static IHostBuilder UseServiceBaseLifetime(this IHostBuilder hostBuilder)
{
return hostBuilder.ConfigureServices((hostContext, services) => services.AddSingleton<IHostLifetime, ServiceBaseLifetime>());
}
public static Task RunAsServiceAsync(this IHostBuilder hostBuilder, CancellationToken cancellationToken = default)
{
return hostBuilder.UseServiceBaseLifetime().Build().RunAsync(cancellationToken);
}
}
ServiceBaseLifetime class to handle service lifecycle
public class ServiceBaseLifetime : ServiceBase, IHostLifetime
{
private readonly TaskCompletionSource<object> _delayStart = new TaskCompletionSource<object>();
public ServiceBaseLifetime(IHostApplicationLifetime applicationLifetime)
{
ApplicationLifetime = applicationLifetime ?? throw new ArgumentNullException(nameof(applicationLifetime));
}
private IHostApplicationLifetime ApplicationLifetime { get; }
public Task WaitForStartAsync(CancellationToken cancellationToken)
{
cancellationToken.Register(() => _delayStart.TrySetCanceled());
ApplicationLifetime.ApplicationStopping.Register(Stop);
new Thread(Run).Start(); // Otherwise this would block and prevent IHost.StartAsync from finishing.
return _delayStart.Task;
}
private void Run()
{
try
{
Run(this); // This blocks until the service is stopped.
_delayStart.TrySetException(new InvalidOperationException("Stopped without starting"));
}
catch (Exception ex)
{
_delayStart.TrySetException(ex);
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
Stop();
return Task.CompletedTask;
}
// Called by base.Run when the service is ready to start.
protected override void OnStart(string[] args)
{
_delayStart.TrySetResult(null);
base.OnStart(args);
}
// Called by base.Stop. This may be called multiple times by service Stop, ApplicationStopping, and StopAsync.
// That's OK because StopApplication uses a CancellationTokenSource and prevents any recursion.
protected override void OnStop()
{
ApplicationLifetime.StopApplication();
base.OnStop();
}
}
The actual implementation of the service is irrelevant other than the constructor, which takes the logger and configuration through DI.
private readonly ILogger<Importer> _logger;
private readonly IConfiguration _configuration;
public Importer(IConfiguration configuration, ILogger<Importer> logger)
{
_logger = logger;
_configuration = configuration;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_logger.LogInformation($"Why is {_configuration["Key1"]} empty?");
}
appsettings.json
{
"Key1":"some value"
}
When i run through debug the console app starts up and runs and logs and has the configuration loaded from appsettings. When i deploy as a service the configuration object is empty.
Notes: The appsettings file is being read, i can tell this by changing the name of it and it throws an exception for file not found. The appsettings file is also not empty.
My issue appears to be some kind of async race condition problem (I am guessing, not positive). The first tick through ExecuteAsync the configuration is not loaded, but the second time through it is. I had the service dying if it encountered that exception, so i never got it to tick a second time.
This appears to be an XY problem and is worthy of a refactor
Create a strongly typed model to hold the desired settings
public class ImporterSettings {
public string Key1 { get; set; }
}
Refactor hosted service to depend on the settings since tightly coupling services to IConfiguration is a code smell in my opinion
private readonly ILogger<Importer> _logger;
private readonly ImporterSettnigs settings;
public Importer(ImporterSettnigs settings, ILogger<Importer> logger) {
_logger = logger;
this.settings = settings;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
_logger.LogInformation($"This is Key1: {settings.Key1}");
}
Now properly configure start up to use the provided configuration
public class Program {
public static async System.Threading.Tasks.Task Main(string[] args) {
var hostbuilder = new HostBuilder()
.UseWindowsService()
.ConfigureAppConfiguration((hostingContext, config) => {
var path = Path.GetDirectoryName(System.Reflection.Assembly.GetEntryAssembly().Location);
config
.SetBasePath(path)
.AddJsonFile("appsettings.json");
})
.ConfigureLogging(
options => options.AddFilter<EventLogLoggerProvider>(level =>
level >= LogLevel.Information)
)
.ConfigureServices((hostContext, services) => {
//get settings from app configuration.
ImporterSettings settings = hostContext.Configuration.Get<ImporterSettings>();
services
.AddSingleton(settings) //add to service collection
.AddHostedService<Importer>()
.Configure<EventLogSettings>(config => {
config.LogName = "Application";
config.SourceName = "Importer";
});
});
#if (DEBUG)
await hostbuilder.RunConsoleAsync();
#else
await hostbuilder..Build().RunAsync();
#endif
}
}

Performing a health check in .NET Core Worker Service

How can I implement health checks in a .NET Core Worker Service?
The service will be run inside Docker and needs to be able to check the health of the service.
Another way of doing this is to implement IHealthCheckPublisher.
The benefits of this approach is the ability to re-use your existing IHealthChecks or integration with 3rd party libraries that rely on IHealthCheck interface (like this one).
Though you still target Microsoft.NET.Sdk.Web as the SDK you don't need to add any asp.net specifics.
Here is an example:
public static IHostBuilder CreateHostBuilder(string[] args)
{
return Host
.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services
.AddHealthChecks()
.AddCheck<RedisHealthCheck>("redis_health_check")
.AddCheck<RfaHealthCheck>("rfa_health_check");
services.AddSingleton<IHealthCheckPublisher, HealthCheckPublisher>();
services.Configure<HealthCheckPublisherOptions>(options =>
{
options.Delay = TimeSpan.FromSeconds(5);
options.Period = TimeSpan.FromSeconds(5);
});
});
}
public class HealthCheckPublisher : IHealthCheckPublisher
{
private readonly string _fileName;
private HealthStatus _prevStatus = HealthStatus.Unhealthy;
public HealthCheckPublisher()
{
_fileName = Environment.GetEnvironmentVariable(EnvVariableNames.DOCKER_HEALTHCHECK_FILEPATH) ??
Path.GetTempFileName();
}
public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
{
// AWS will check if the file exists inside of the container with the command
// test -f $DOCKER_HEALTH_CHECK_FILEPATH
var fileExists = _prevStatus == HealthStatus.Healthy;
if (report.Status == HealthStatus.Healthy)
{
if (!fileExists)
{
using var _ = File.Create(_fileName);
}
}
else if (fileExists)
{
File.Delete(_fileName);
}
_prevStatus = report.Status;
return Task.CompletedTask;
}
}
I don't think is worth it to change SDK to Microsoft.NET.Sdk.Web. You will include additional middlewares just because of one health check? No thanks ...
What you could do is to use a different protocol like TCP.
The general idea is:
Create a separate background service that creates a TCP server (take a look at TcpListener.cs)
When you receive a request you have two options: if the application is healthy accept TCP connection otherwise reject it.
If you use containers your orchestrator should have an option to call it over TCP (in k8s there is a property tcpSocket)
If you need more detailed information you may check: Monitoring Health of ASP.NET Core Background Services With TCP Probes on Kubernetes
Cheers!
I think that you should also consider to retain the Microsoft.NET.Sdk.Worker.
Don't change the whole sdk just because of the health checks.
Then you can create a backgroundservice (just like the main worker), in order to update a file to write for example the current timestamp. An example of the background health check worker would be:
public class HealthCheckWorker : BackgroundService
{
private readonly int _intervalSec;
private readonly string _healthCheckFileName;
public HealthCheckWorker(string healthCheckFileName, int intervalSec)
{
this._intervalSec = intervalSec;
this._healthCheckFileName = healthCheckFileName;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (true)
{
File.WriteAllText(this._healthCheckFileName, DateTime.UtcNow.ToString());
await Task.Delay(this._intervalSec * 1000, stoppingToken);
}
}
}
Then you can add a extension method like this:
public static class HealthCheckWorkerExtensions
{
public static void AddHealthCheck(this IServiceCollection services,
string healthCheckFileName, int intervalSec)
{
services.AddHostedService<HealthCheckWorker>(x => new HealthCheckWorker(healthCheckFileName, intervalSec));
}
}
With this you can add in services the health check support
.ConfigureServices(services =>
{
services.AddHealthCheck("hc.txt", 5);
})
Add HTTPListener and expose the health checks endpoints.
Using HTTPListener does not require adding Microsoft.NET.Sdk.Web SDK.
Program.cs
using Consumer;
IHost host = Host.CreateDefaultBuilder(args)
.ConfigureServices(services =>
{
services.AddHostedService<Worker>();
services.AddHostedService<HttpHealthcheck>();
})
.Build();
await host.RunAsync();
HttpHealthcheck.cs
using System.Net;
using System.Text;
namespace Consumer;
public class HttpHealthcheck : BackgroundService
{
private readonly ILogger<Worker> _logger;
private readonly HttpListener _httpListener;
private readonly IConfiguration _configuration;
public HealthcheckHttpListener(ILogger<Worker> logger, IConfiguration configuration)
{
_logger = logger;
_configuration = configuration;
_httpListener = new HttpListener();
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
_httpListener.Prefixes.Add($"http://*:5001/healthz/live/");
_httpListener.Prefixes.Add($"http://*:5001/healthz/ready/");
_httpListener.Start();
_logger.LogInformation($"Healthcheck listening...");
while (!stoppingToken.IsCancellationRequested)
{
HttpListenerContext ctx = null;
try
{
ctx = await _httpListener.GetContextAsync();
}
catch (HttpListenerException ex)
{
if (ex.ErrorCode == 995) return;
}
if (ctx == null) continue;
var response = ctx.Response;
response.ContentType = "text/plain";
response.Headers.Add(HttpResponseHeader.CacheControl, "no-store, no-cache");
response.StatusCode = (int)HttpStatusCode.OK;
var messageBytes = Encoding.UTF8.GetBytes("Healthy");
response.ContentLength64 = messageBytes.Length;
await response.OutputStream.WriteAsync(messageBytes, 0, messageBytes.Length);
response.OutputStream.Close();
response.Close();
}
}
}
What I've done to accomplish this is add Microsoft.NET.Sdk.Web to my Worker, and then configured a web host to run alongside the worker:
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(builder =>
{
builder.UseStartup<Startup>();
})
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Worker>();
services.AddLogging(builder =>
builder
.AddDebug()
.AddConsole()
);
});
With that done, all that's left to do is map the health check endpoint as you normally would with ASP.NET Core.

Categories