We Keep Coding

C# error propagating up the stack when it should not

I'm encountering an issue where a service is exiting on errors that should never propagate up.
I built a microservice manager (.NET as the local environment doesnt support .NET Core and some of its native microservice abilities)
Built in VS2019 targeting .NET 4.5.2 (I know, but this is the world we live in)
For the microservice manager, it is built and installed as a windows service. Entry looks like this (#if/#else was for testing locally, it is working as intended when registered as a windows service)
Program.cs (Entry point)
` static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
#if DEBUG
Scheduler myScheduler = new Scheduler();
myScheduler.OnDebug();
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
#else
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Scheduler()
};
ServiceBase.Run(ServicesToRun);
#endif
}
}`
Scheduler.cs
//(confidential code hidden)
`private static readonly Configuration config = Newtonsoft.Json.JsonConvert.DeserializeObject<Configuration>(
File.ReadAllText(configFilePath)
);
public Scheduler()
{
//InitializeComponent(); //windows service, doesnt need UI components initialized
}
public void OnDebug()
{
OnStart(null); //triggers when developing locally
}
protected override async void OnStart(string[] args)
{
try
{
logger.Log($#"Service manager starting...");
logger.Log($#"Finding external services... {config.services.Count} services found.");
foreach (var service in config.services)
{
try
{
if (service.disabled)
{
logger.Log(
$#"Skipping {service.name}: disabled=true in Data Transport Service's appSettings.json file");
continue;
}
logger.Queue($#"Starting: {service.name}...");
string serviceLocation = service.useRelativePath
? Path.Combine(assemblyLocation, service.path)
: service.path;
var svc = Assembly.LoadFrom(serviceLocation);
var assemblyType = svc.GetType($#"{svc.GetName().Name}.Program");
var methodInfo = assemblyType.GetMethod("Main");
var instanceObject = Activator.CreateInstance(assemblyType, new object[0]);
methodInfo.Invoke(instanceObject, new object[0]);
logger.Queue(" Running").Send("");
}
catch (TargetInvocationException ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
catch (Exception ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
}
logger.Log("Finished loading services.");
}
catch (Exception ex)
{
logger.Log($#"Critical error encountered", LOG.LEVEL.CRITICAL, ex);
}
}
Microservice:
public [Confidential]()
{
if (currentProfile == null)
{
var errMsg =
$#"Service not loaded, Profile not found, check appSettings.currentProfile: '{config.currentProfile}'";
logger.Log(errMsg,severity: LOG.LEVEL.CRITICAL);
throw new SettingsPropertyNotFoundException(errMsg);
}
if (currentProfile.disabled)
{
var errMsg = $#"Service not loaded: {config.serviceName}, Service's appSettings.currentProfile.disabled=true";
logger.Log(errMsg,LOG.LEVEL.WARN);
throw new ArgumentException(errMsg);
}
logger.Log($#"Loading: '{config.serviceName}' with following configuration:{Environment.NewLine}{JsonConvert.SerializeObject(currentProfile,Formatting.Indented)}");
logger.Queue($#"Encrypting config file passwords...");
bool updateConfig = false;
foreach (var kafkaSource in config.dataTargets)
{
if (!kafkaSource.password.IsEncrypted())
{
updateConfig = true;
logger.Queue($#"%tabEncrypting: {kafkaSource.name}");
kafkaSource.password = kafkaSource.password.Encrypt();
}
else
{
logger.Queue($#"%tabAlready encrypted: {kafkaSource.name}");
}
}
logger.Send(Environment.NewLine);
if (updateConfig)
{
File.WriteAllText(
configFilePath,
Newtonsoft.Json.JsonConvert.SerializeObject(config));
}
var _source = config.dataSources.FirstOrDefault(x=>x.name==currentProfile.dataSource);
var _target = config.dataTargets.FirstOrDefault(x => x.name == currentProfile.dataTarget);
source = new Connectors.Sql(logger,
_source?.name,
_source?.connectionString,
_source.pollingInterval,
_source.maxRowsPerSelect,
_source.maxRowsPerUpdate);
target = new Connectors.KafkaProducer(logger)
{
bootstrapServers = _target?.bootstrapServers,
name = _target?.name,
password = _target?.password.Decrypt(),
sslCaLocation = Path.Combine(assemblyLocation,_target?.sslCaLocation),
topic = _target?.topic,
username = _target?.username
};
Start();
}
public void Start()
{
Timer timer = new Timer();
try
{
logger.Log($#"SQL polling interval: {source.pollingInterval} seconds");
timer.Interval = source.pollingInterval * 1000;
timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
timer.Start();
if (currentProfile.executeOnStartup)
Run();
}
catch (Exception ex)
{
var sb = new StringBuilder();
sb.AppendLine($#"Critical error encountered loading external service: {config.serviceName}.");
if (!timer.Enabled)
sb.AppendLine($#"service unloaded - Schedule not started!");
else
sb.AppendLine($#"service appears to be loaded and running on schedule.");
logger.Log(sb.ToString(), LOG.LEVEL.CRITICAL, ex);
}
}
public void OnTimer(object sender, ElapsedEventArgs e)
{
try
{
Run();
}
catch (Exception ex)
{
logger.Log($#"Critical error during scheduled run on service: {config.serviceName}.", LOG.LEVEL.CRITICAL, ex);
}
}
public async void Run()
{
//Get new alarm events from SQL source
logger.Queue("Looking for new alarms...");
var rows = await GetNewEvents();`
The exception occurred during the GetNewEvents method, which attempted to open a SqlConnection to a SQL server that was unavailable due to network issues, that method intentionally throws an exception, which should throw up to OnTimer, where it gets caught, logged, and the timer keeps running. During development/testing, I used invalid credentials, bad connection string, etc and simulated this type of error and it worked as expected, logged the error, kept running. For some reason recently, that error is not caught in OnTimer, it propagates up, where it should be caught by Start (but isn't), after that it should be caught by the parent service manager which is entirely wrapped in a try/catch with no throw's, and above that (because their could be multiple microservices managed by that service) the entry point to the service manager is wrapped in try/catch with no throws, all for isolation from microservice errors. For some reason though, now, the error from a VERY downstream application is propagating all the way up.
Typically, this code runs 24/7 no issues, the microservice it is loading from the config file launches and runs fine. The entry into that specific microservice starts with a try {...} catch (Exception ex) {...} block.
The concept is to have a microservice manager than can launch a number of microservices without having to install all of them as windows services, and have some level of configuration driven by a config file that dictates how the main service runs.
The microservice represented here opens a SQL connection, reads data, performs business logic, publishes results to Kafka, it does this on a polling interval dictated by the config file contained in the microservice. As stated above, its ran for months without issue.
Recently, I noticed the main microservice manager service was not running on the windows server, I investigated the Server Application Logs and found a "Runtime Error" that essentially stated the microservice, while attempting to connect to sql, failed (network issue) and caused the entire microservice manager to exit. To my understanding, they way I'm launching the microservice should isolate it from the main service manager app. Additionally, the main service manager app is wrapped in a very generic try catch block. The entry point to the micro service itself is wrapped in a try catch, and almost every component in the microservice is wrapped in try / catch per business need. The scenario that faulted (cant connect to sql) intentionally throws an error for logging purposes, but should be caught by the immediate parent try/catch, which does not propagate or re-throw, only logs the error to a txt file and the windows server app log.
How is it that this exception is bubbling up through isolation points and causing the main service to fault and exit? I tested this extensively during development and prior to release, this exact scenario being unable to connect to sql, and it generated the correct log entry, and tried again on the next polling cycle as expected.
I haven't tried any other approaches as yet, as I feel they would be band-aid fixes as best as I dont understand why the original design is suddenly failing. The server hasn't changed, no patching/security updates/etc.
From the server Application Log:
Application: DataTransportService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
at Connectors.SqlHelper.DbHelper+d__13`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__14.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

WindowsService FileSystemWatcher Crash on InputOnChanged call

I have a Windows Service that I successfully deploys, successfully works when debugging, but crashes when a file is added to the monitored directory.
I thought it was an issue with my impersonator being used between the OnStart and InputOnChanged, but the crash still happens when I run the service under my own domain user.
I have EventLog set to write to it's own application source, but none of my WriteEntrys are called except the one in the OnStart function. I've been trying different tweaks and feel like I need another set of eyes to see something i'm not:
protected override void OnStart(string[] args)
{
//using(Impersonator context = new Impersonator("XXXXX", "XXXXXXXX", "XXXXXXXXXX"))
//{
try
{
this.fileWatcherService = new FileSystemWatcher(baseFilePath, "*.txt")
{
NotifyFilter = NotifyFilters.LastWrite
};
fileWatcherService.Changed += InputOnChanged;
fileWatcherService.EnableRaisingEvents = true;
eventLog.WriteEntry("XXXX-XXXXX-Service Started");
}
catch (Exception ex)
{
eventLog.WriteEntry($"{baseFilePath} was not accessible to monitor because {ex.Message}", EventLogEntryType.Error);
}
}
protected void InputOnChanged(object source, FileSystemEventArgs e)
{
if (e.ChangeType == WatcherChangeTypes.Changed)
{
eventLog.WriteEntry($"Change Detected - File {e.Name}", EventLogEntryType.Information);
try
{
fileWatcherService.EnableRaisingEvents = false;
eventLog.WriteEntry("Starting process for file: " + e.Name);
if (!File.Exists(e.FullPath))
{
eventLog.WriteEntry($"{e.Name} was not accessible", EventLogEntryType.Error);
}
//Copy File to backup copy before formatting
File.Copy(e.FullPath
, Path.Combine(#"\\XXXXXX\XXXXXX\XXXXXXXX\XXXXXXXXXX XXX XXXXXXXXX\XXX\XXXX\XXXX\BackupFiles", GetBackupFileName(e.Name))
, false);
//Save formatted file to directory
List<string> lines = System.IO.File.ReadAllLines(e.FullPath).ToList();
File.WriteAllText(Path.Combine(#"\\XXXXXX\XXXXXX\XXXXXXXX\XXXXXXXXXX XXX XXXXXXXXX\XXX\XXXX\XXXX\FormattedFiles", GetFormattedFileName(e.Name))
, CSVFormatService.FormatLines(lines));
//Remove file from base path to prevent re-processing
File.Delete(e.FullPath);
eventLog.WriteEntry($"Successfully moved {e.FullPath}", EventLogEntryType.Information);
}
catch (Exception ex)
{
eventLog.WriteEntry("XXXX-XXXXX-Service exception: " + ex.Message, EventLogEntryType.Error);
}
finally
{
fileWatcherService.EnableRaisingEvents = true;
}
}
}
Would expect eventLog.WriteEntry("Starting process for file: " + e.Name); to update the Application log at least because that is before any attempt to touch a file, but I don't see that in the log. However, the service runs until I place a test file in the monitored directory, and then crashes with a unhandled exception of file does not exist

When building out these services, make sure you reference a shared project correctly. This issue was caused by adding a reference to a class library to the project, but the .dll was missing when deploying the service. So when the service tried to access the .dll to process data a FileNotFound exception was being thrown. This also make sense as to why the exception was marked as unhandled.

How to re-start a console application if it crashes?

I have created a console application in C#. How can I program this application so that it will re-start itself after a crash?

If I understand your question correctly, you want to attempt to re-start a console app in the event of a crash. In C# console-apps the method defined as the entry point (usually static void main) is the root of the call stacks in the app. You essentially would need to call that method recursively. You will want to make sure that the app eventually fails if it is in some unintended or unrecoverable state.
For example in the main class:
static int retryCount;
const int numberOfRetries = 3;
static void Main(string[] args)
{
try
{
var theApp = new MyApplicationType(args);
theApp.StartMyAppLogic();
}
catch (ExpectedExceptionType expectThisTypeOfException)
{
thisMethodHandlesExceptions(expectThisTypeOfException);
}
catch (AnotherExpectedExceptionType alsoExpectThisTypeOfException)
{
thisMethodHandlesExceptions(alsoExpectThisTypeOfException);
}
catch (Exception unexpectedException)
{
if(retryCount < numberOfRetries)
{
retryCount++;
Main(args);
}
else
{
throw;
}
}
}

You can use a watchdog to process your monitor and restart it if crashed:
see: What's the best way to watchdog a desktop application?
You can use a windows service instead and set it's recovery options as indicated here: https://serverfault.com/questions/48600/how-can-i-automatically-restart-a-windows-service-if-it-crashes
You can use a scheduled task in task manager to start your application periodically , and set it to only start if previous run has ended:
https://support.microsoft.com/en-us/kb/323527

You could try something like this:
static void Main(string[] args)
{
try
{
// Application code goes here
}
catch (Exception)
{
var applicationPath = System.Reflection.Assembly.GetExecutingAssembly().Location;
Process.Start(applicationPath);
Environment.Exit(Environment.ExitCode);
}
}
Basically, wrap all the code in a try/catch, and if any exceptions occur, the program will retrieve the .exe location with System.Reflection.Assembly.GetExecutingAssembly().Location; and then call Process.Start to run the application again.

You should control your console app from another application (watchdog, sheduler, procmon, servman, ...).
E.g. you can create your console app as a service and control it from service manager.

Handling Global Exception Xamarin | Droid | iOS

We all know that mobile is compact platform where we have to look lots of things while building an application. It could be anything e.g. Memory Performance Resolutions Architecture Implementation etc. We never know when and what causes app crash a big ISSUE while playing with the app, It could happen anytime
e.g. App Launch, Load Screen, API Call, Binding Data, Loading Images etc.
And trust me sometime its really hard to find where and what cause an issue in app. I saw many post on forums, tech community and groups which is related to the same issue, where peoples usually asking questions as:
App Crashing at launching.
App Crash at Splash Screen loading.
App Crash while Image showing.
App Crashing while binding data from api.
How to identify issue and where it causes?

Purpose: Our purpose here to grab an exception's stack trace data that help us to identify what exactly causes the issue whether in Release Mode or Debug Mode. We will be able to understand the issue and the reason that causes it. We will store this data in a text file that will be store in device storage.
Solution: Alternatively you can make your own insight grabber that will give you you app insight and clue if something went wrong while testing the app. Its will be your, you can tweak like you want. let's dive to try{} and catch{} globally.
Create a Helper Class file that has a method to generate a Text file for exception data.
public static class ExceptionFileWriter
{
#region Property File Path
static string FilePath
{
get
{
string path = string.Empty;
var _fileName = "Fatal.txt";
#if __IOS__
string documentsPath = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments); // Documents folder C:\ddddd
string libraryPath = Path.Combine(documentsPath, "..", "Library"); // Library folder C:\dddd\...\library
path = Path.Combine(libraryPath, _fileName); //c:\ddddd\...\library\NBCCSeva.db3
#else
#if __ANDROID__
string dir = Path.Combine(Android.OS.Environment.ExternalStorageDirectory.ToString(), "Exception");
if (Directory.Exists(dir))
return Path.Combine(dir, _fileName);
path= Path.Combine(Directory.CreateDirectory(dir).FullName, _fileName);
#endif
#endif
return path;
}
}
#endregion
#region ToLog Exception
public static void ToLogUnhandledException(this Exception exception)
{
try
{
var errorMessage = String.Format("Time: {0}\r\nError: Unhandled Exception\r\n{1}\n\n", DateTime.Now, string.IsNullOrEmpty(exception.StackTrace) ? exception.ToString() : exception.StackTrace);
File.WriteAllText(FilePath, errorMessage);
}
catch (Exception ex)
{
// just suppress any error logging exceptions
}
}
#endregion
}
Time to implement code: Subscribe following events inside your app's Application file or Splash Activity. I'm using Application in this case.
AppDomain.CurrentDomain.UnhandledException += CurrentDomainOnUnhandledException;
TaskScheduler.UnobservedTaskException += TaskSchedulerOnUnobservedTaskException;
[Application]
public class ExceptionHandlingApp : Application
{
#region Constructor
public ExceptionHandlingApp(IntPtr javaReference, JniHandleOwnership transfer)
: base(javaReference, transfer)
{
}
#endregion
#region OnCreate
public override void OnCreate()
{
base.OnCreate();
AppDomain.CurrentDomain.UnhandledException += CurrentDomainOnUnhandledException;
TaskScheduler.UnobservedTaskException += TaskSchedulerOnUnobservedTaskException;
}
#endregion
#region Task Schedular Exception
private static void TaskSchedulerOnUnobservedTaskException(object sender, UnobservedTaskExceptionEventArgs unobservedTaskExceptionEventArgs)
{
var newExc = new Exception("TaskSchedulerOnUnobservedTaskException", unobservedTaskExceptionEventArgs.Exception);
newExc.ToLogUnhandledException();
}
#endregion
#region Current Domain Exception
private static void CurrentDomainOnUnhandledException(object sender, UnhandledExceptionEventArgs unhandledExceptionEventArgs)
{
var newExc = new Exception("CurrentDomainOnUnhandledException", unhandledExceptionEventArgs.ExceptionObject as Exception);
newExc.ToLogUnhandledException();
}
#endregion
}
Note: You can find exceptions record file in Device Storage | File Manager > Exception Folder > fatal.txt
Cheers!!

Beside of doing it on your own you can also use Xamarin.Insights as it is free to use for Xamarin users and has got implementations for all platforms.
You receive usage reports, crash reports etc. online without the need for the user to send you a log file manually.
The only thing you need to do to receive crash reports is to initialize Xamarin.Insights on startup of your app:
Insights.HasPendingCrashReport += (sender, isStartupCrash) =>
{
if (isStartupCrash) {
Insights.PurgePendingCrashReports().Wait();
}
};
Insights.Initialize("Your API Key");

Unable to catch ServiceController.Start() exceptions in calling thread

How can i catch the exception that occurs when starting a windows service. I am unable to get the exception here in my below code even though i am throwing exception in the Onstart() method of the service.
public class InterOpIntegrationWinService : ServiceBase
{
protected override void OnStart(string[] args)
{
throw new InvalidOperationException(message);
}
}
Calling thread code
try
{
using (ServiceController controller = new ServiceController())
{
controller.ServiceName = objServiceConfig.ServiceName;
controller.Start();
System.Windows.Forms.Application.DoEvents();
//controller.WaitForStatus(ServiceControllerStatus.Running, new TimeSpan(0, 0, 15));
//controller.WaitForStatus(ServiceControllerStatus.Running);
//if (!string.IsNullOrEmpty(LogUtilities.ServiceOnStartException))
//{
// MessageBox.Show("Error with starting service : " + LogUtilities.ServiceOnStartException);
// LogUtilities.ServiceOnStartException = string.Empty;
//}
}
}
catch (System.InvalidOperationException InvOpExcep)
{
DisplayError(InvOpExcep.Message);
LogUtilities.DisplayMessage("Failed to start service. " + LogUtilities.ServiceOnStartException, InvOpExcep);
LogUtilities.ServiceOnStartException = string.Empty;
}
catch (Exception ex)
{
DisplayError(ex.Message);
LogUtilities.DisplayMessage("Failed to start service. " + LogUtilities.ServiceOnStartException, ex);
LogUtilities.ServiceOnStartException = string.Empty;
}
i check for application license in the onstart() method and throws a licensing error if it fails. i want this to shared to my calling thread so i could show the message in a DialogBox. Any ideas of how to do this if i cannot handle the exceptions in my calling process.

Separate your service into (at least) two components - a component that deals with IPC in some form (e.g. Remoting, WCF endpoint, REST service, etc) and (one or more) components that do its actual job.
If the licensing check fails, don't start the other components - but do still start the component that offers IPC. After starting your service (which should now always at least start), you forms-based application can connect to the service and (through whatever means you want) determine that the service is currently refusing to provide any functionality due to a failed licensing check.