We Keep Coding

C# error propagating up the stack when it should not

I'm encountering an issue where a service is exiting on errors that should never propagate up.
I built a microservice manager (.NET as the local environment doesnt support .NET Core and some of its native microservice abilities)
Built in VS2019 targeting .NET 4.5.2 (I know, but this is the world we live in)
For the microservice manager, it is built and installed as a windows service. Entry looks like this (#if/#else was for testing locally, it is working as intended when registered as a windows service)
Program.cs (Entry point)
` static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
#if DEBUG
Scheduler myScheduler = new Scheduler();
myScheduler.OnDebug();
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
#else
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Scheduler()
};
ServiceBase.Run(ServicesToRun);
#endif
}
}`
Scheduler.cs
//(confidential code hidden)
`private static readonly Configuration config = Newtonsoft.Json.JsonConvert.DeserializeObject<Configuration>(
File.ReadAllText(configFilePath)
);
public Scheduler()
{
//InitializeComponent(); //windows service, doesnt need UI components initialized
}
public void OnDebug()
{
OnStart(null); //triggers when developing locally
}
protected override async void OnStart(string[] args)
{
try
{
logger.Log($#"Service manager starting...");
logger.Log($#"Finding external services... {config.services.Count} services found.");
foreach (var service in config.services)
{
try
{
if (service.disabled)
{
logger.Log(
$#"Skipping {service.name}: disabled=true in Data Transport Service's appSettings.json file");
continue;
}
logger.Queue($#"Starting: {service.name}...");
string serviceLocation = service.useRelativePath
? Path.Combine(assemblyLocation, service.path)
: service.path;
var svc = Assembly.LoadFrom(serviceLocation);
var assemblyType = svc.GetType($#"{svc.GetName().Name}.Program");
var methodInfo = assemblyType.GetMethod("Main");
var instanceObject = Activator.CreateInstance(assemblyType, new object[0]);
methodInfo.Invoke(instanceObject, new object[0]);
logger.Queue(" Running").Send("");
}
catch (TargetInvocationException ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
catch (Exception ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
}
logger.Log("Finished loading services.");
}
catch (Exception ex)
{
logger.Log($#"Critical error encountered", LOG.LEVEL.CRITICAL, ex);
}
}
Microservice:
public [Confidential]()
{
if (currentProfile == null)
{
var errMsg =
$#"Service not loaded, Profile not found, check appSettings.currentProfile: '{config.currentProfile}'";
logger.Log(errMsg,severity: LOG.LEVEL.CRITICAL);
throw new SettingsPropertyNotFoundException(errMsg);
}
if (currentProfile.disabled)
{
var errMsg = $#"Service not loaded: {config.serviceName}, Service's appSettings.currentProfile.disabled=true";
logger.Log(errMsg,LOG.LEVEL.WARN);
throw new ArgumentException(errMsg);
}
logger.Log($#"Loading: '{config.serviceName}' with following configuration:{Environment.NewLine}{JsonConvert.SerializeObject(currentProfile,Formatting.Indented)}");
logger.Queue($#"Encrypting config file passwords...");
bool updateConfig = false;
foreach (var kafkaSource in config.dataTargets)
{
if (!kafkaSource.password.IsEncrypted())
{
updateConfig = true;
logger.Queue($#"%tabEncrypting: {kafkaSource.name}");
kafkaSource.password = kafkaSource.password.Encrypt();
}
else
{
logger.Queue($#"%tabAlready encrypted: {kafkaSource.name}");
}
}
logger.Send(Environment.NewLine);
if (updateConfig)
{
File.WriteAllText(
configFilePath,
Newtonsoft.Json.JsonConvert.SerializeObject(config));
}
var _source = config.dataSources.FirstOrDefault(x=>x.name==currentProfile.dataSource);
var _target = config.dataTargets.FirstOrDefault(x => x.name == currentProfile.dataTarget);
source = new Connectors.Sql(logger,
_source?.name,
_source?.connectionString,
_source.pollingInterval,
_source.maxRowsPerSelect,
_source.maxRowsPerUpdate);
target = new Connectors.KafkaProducer(logger)
{
bootstrapServers = _target?.bootstrapServers,
name = _target?.name,
password = _target?.password.Decrypt(),
sslCaLocation = Path.Combine(assemblyLocation,_target?.sslCaLocation),
topic = _target?.topic,
username = _target?.username
};
Start();
}
public void Start()
{
Timer timer = new Timer();
try
{
logger.Log($#"SQL polling interval: {source.pollingInterval} seconds");
timer.Interval = source.pollingInterval * 1000;
timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
timer.Start();
if (currentProfile.executeOnStartup)
Run();
}
catch (Exception ex)
{
var sb = new StringBuilder();
sb.AppendLine($#"Critical error encountered loading external service: {config.serviceName}.");
if (!timer.Enabled)
sb.AppendLine($#"service unloaded - Schedule not started!");
else
sb.AppendLine($#"service appears to be loaded and running on schedule.");
logger.Log(sb.ToString(), LOG.LEVEL.CRITICAL, ex);
}
}
public void OnTimer(object sender, ElapsedEventArgs e)
{
try
{
Run();
}
catch (Exception ex)
{
logger.Log($#"Critical error during scheduled run on service: {config.serviceName}.", LOG.LEVEL.CRITICAL, ex);
}
}
public async void Run()
{
//Get new alarm events from SQL source
logger.Queue("Looking for new alarms...");
var rows = await GetNewEvents();`
The exception occurred during the GetNewEvents method, which attempted to open a SqlConnection to a SQL server that was unavailable due to network issues, that method intentionally throws an exception, which should throw up to OnTimer, where it gets caught, logged, and the timer keeps running. During development/testing, I used invalid credentials, bad connection string, etc and simulated this type of error and it worked as expected, logged the error, kept running. For some reason recently, that error is not caught in OnTimer, it propagates up, where it should be caught by Start (but isn't), after that it should be caught by the parent service manager which is entirely wrapped in a try/catch with no throw's, and above that (because their could be multiple microservices managed by that service) the entry point to the service manager is wrapped in try/catch with no throws, all for isolation from microservice errors. For some reason though, now, the error from a VERY downstream application is propagating all the way up.
Typically, this code runs 24/7 no issues, the microservice it is loading from the config file launches and runs fine. The entry into that specific microservice starts with a try {...} catch (Exception ex) {...} block.
The concept is to have a microservice manager than can launch a number of microservices without having to install all of them as windows services, and have some level of configuration driven by a config file that dictates how the main service runs.
The microservice represented here opens a SQL connection, reads data, performs business logic, publishes results to Kafka, it does this on a polling interval dictated by the config file contained in the microservice. As stated above, its ran for months without issue.
Recently, I noticed the main microservice manager service was not running on the windows server, I investigated the Server Application Logs and found a "Runtime Error" that essentially stated the microservice, while attempting to connect to sql, failed (network issue) and caused the entire microservice manager to exit. To my understanding, they way I'm launching the microservice should isolate it from the main service manager app. Additionally, the main service manager app is wrapped in a very generic try catch block. The entry point to the micro service itself is wrapped in a try catch, and almost every component in the microservice is wrapped in try / catch per business need. The scenario that faulted (cant connect to sql) intentionally throws an error for logging purposes, but should be caught by the immediate parent try/catch, which does not propagate or re-throw, only logs the error to a txt file and the windows server app log.
How is it that this exception is bubbling up through isolation points and causing the main service to fault and exit? I tested this extensively during development and prior to release, this exact scenario being unable to connect to sql, and it generated the correct log entry, and tried again on the next polling cycle as expected.
I haven't tried any other approaches as yet, as I feel they would be band-aid fixes as best as I dont understand why the original design is suddenly failing. The server hasn't changed, no patching/security updates/etc.
From the server Application Log:
Application: DataTransportService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
at Connectors.SqlHelper.DbHelper+d__13`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__14.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

System.Diagnostics.Process.HasExited always returns true and Process.Exited event handler always hits when process is still open

I am currently having issues with a piece of my code. This code works perfectly fine (even without some of the extra if statements) on my machine when I run it both through Visual Studio 2013 and when I publish using ClickOnce. On the client's machine, the event handler for the process exiting catches, the process ID is correct, and the process.hasExited returns true all while the process is still open. The process opens a pdf for them to sign and waits for them to save it and close it before proceeding. It uses Adobe Acrobat Reader DC.
The error message received is from the catch block of the event handler method:
System.IO.IOException: The process cannot access the file because it is being used by another process.
This occurs at the call for getReaders_NewHire(emp).
This is the method that opens the process:
private void pdfAndEmail(Employee employee, Requirements requirements)
{
try
{
PDFUtility pdfu = new PDFUtility();
pdfu.createPDFMG_NewHire(employee, requirements);
emp = employee;
process = new System.Diagnostics.Process();
string path = System.IO.Path.GetTempPath() + "\\ITResources\\Manager\\NewHire_" + employee.m_name.Replace(" ", "") + ".pdf";
Uri pdf = new Uri(path, UriKind.RelativeOrAbsolute);
process.StartInfo.FileName = pdf.LocalPath;
process.EnableRaisingEvents = true;
process.Exited += new EventHandler(process_Exited);
process.Start();
pid = process.Id;
}
catch (Exception e)
{
MessageBox.Show(e.ToString(), "Error Message", MessageBoxButton.OK, MessageBoxImage.Error);
}
}
and this is the event handling method:
private void process_Exited(object sender, EventArgs e)
{
try
{
if (process.Id == pid)
{
if (process.HasExited)
{
PDFUtility pdfu = new PDFUtility();
pdfu.getReaders_NewHire(emp);
Emailer send = new Emailer();
send.SendAfterMG_NewHire();
}
else
MessageBox.Show("Process has not exited.", "Error", MessageBoxButton.OK, MessageBoxImage.Error);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString(), "Error Message", MessageBoxButton.OK, MessageBoxImage.Error);
}
}
The getReaders method in the event handler needs to read the file that the client has to sign so it must be closed beforehand. I cannot force a close because I will never know how long it will take them to sign and I can't have the email being sent out before they've signed and saved the pdf.
I've already tried using the WaitForExit() method and it skips completely (only on my client's computer). I am running Windows 10, but the client is running Windows 7. I have not been able to find any documentation about these methods not working on Win7.
Note: I understand that the if statements in the event handler method are a bit redundant, but I was desperate to find where it was catching.
Any help would be greatly appreciated.
FIXED: I ended up have a Message Box pop up after my PDF was signed to assist the WaitForExit() in the Background Worker thread. After this message box's OK button is pressed (because they pause the application and wait for a response), it then reads the pdf files.

It uses Adobe Acrobat Reader DC.
This is your advantage I think, and my approach assumes this. Consider the following:
private static void Main()
{
var process = Process.GetProcessesByName("AcroRd32").FirstOrDefault();
if (process != null)
{
Process.Start(#"C:\mvvm.pdf");
}
else
{
process = Process.Start(#"C:\mvvm.pdf");
}
process.EnableRaisingEvents = true;
process.Exited += (sender, args) =>
{
Console.WriteLine("Exited");
};
Console.ReadLine();
}
First we check if Adobe process is already running, if so we keep the reference and hook the event handler to that existing process. If not we do what you have already done. Some validation code is omitted.
I have tested with the process running and not running and it works.
NOTICE: It will not work as expected if the user has another application configured as a predefined PDF reader

Handling AppDomain.CurrentDomain.UnhandledException in windows service

I have window service which acts as a sync software. I want to add unhanded exception logging on my service, so I modified my program.cs like this:
static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[SecurityPermission(SecurityAction.Demand, Flags = SecurityPermissionFlag.ControlAppDomain)]
static void Main()
{
// Register Unhandled Exception Handler
AppDomain.CurrentDomain.UnhandledException +=
new UnhandledExceptionEventHandler(UnhandledExceptionHandler);
// Run Service
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Service()
};
ServiceBase.Run(ServicesToRun);
}
static void UnhandledExceptionHandler(object sender, UnhandledExceptionEventArgs args)
{
// Get Exception
Exception ex = (Exception)args.ExceptionObject;
// Generate Error
string ErrorMessage = String.Format(
"Error: {0}\r\n" +
"Runtime Terminating: {1}\r\n----- ----- ----- ----- ----- -----\r\n\r\n" +
"{2}\r\n\r\n####################################\r\n",
ex.Message,
args.IsTerminating,
ex.StackTrace.Trim());
// Write Error To File
try
{
using (StreamWriter sw = File.AppendText("UnhandledExceptions.log"))
sw.WriteLine(errorMessage);
}
catch { }
}
}
Then on my Service.cs file, in the OnStart method, I added a throw new Exception("test"); to see if unhanded exceptions are being logged to file as expected.
When I start my service, it stops immediately as expected; however it doesn't seem to be logging the exception to the specified file.
Any idea what I am doing wrong here? Thanks in advance for any help.
Before you ask, my service runs as Local Service and the directory where my service .exe runs from (c:\mysync) already has Local Service added in the security tab with full read/write access.

OnStart is called in Service base class inside try-catch block. If an exception happens on this stage it catches it and just set a status 1 as a result and do not throw it further:
string[] args = (string[]) state;
try
{
this.OnStart(args);
.....
}
catch (Exception ex)
{
this.WriteEventLogEntry(Res.GetString("StartFailed", new object[1]
{
(object) ((object) ex).ToString()
}), EventLogEntryType.Error);
this.status.currentState = 1;
}
As a result you can find a record in EventLogs, but you can't catch it as an unhanded domain exception, as there is no such exception.

using (StreamWriter sw = File.AppendText("UnhandledExceptions.log"))
It is forever a really bad idea to not use full path names for files (like c:\foo\bar.log). Especially in a service, you have very little control over the default directory for your service. Because it is started by the service control manager, not by the user from the command prompt or a desktop shortcut.
So high odds that you are just looking at the wrong file. The real one probably ended up being written to c:\windows\system32 (or syswow64). The operating system directories are normally write protected but that doesn't work for a service, they run with a highly privileged account so can litter the hard drive anywhere.
Always use full path names. Using the EventLog instead is highly recommended.

.Net Runtime Error Stopping Service; Unhandled IOException

I have been put in charge of updating a service that reads a text file and creates pdf files from sections of the text file and emails out the pdfs. I recently made some changes to the service and used the .Net 4.0 Framework. When updating on the server, the 4.0 Framework was installed before I could move my files and start the service successfully - it was using 2.0 previously. The service runs until it reaches code that attempts to clean up the directory with the pdf files. The service Stops when the code attempts to delete a pdf file which is "in use by another process". The code looks for this exception and is supposed to wait for about 30 seconds and try the delete again. Locally, I run this service through a test Harness and it loops through until the file becomes available to delete (on average it takes about 5 minutes), but on the server the Service stops with the Unhandled IOException found in the Application Event log. I don't understand why it will continue processing locally but not on the server. Any ideas or other information would greatly be appreciated.
Here is the error in the Event Log:
The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException
Stack:
at System.IO.__Error.WinIOError(Int32, System.String)
at System.Console.GetBufferInfo(Boolean, Boolean ByRef)
at processorName.FileProcessingThread.CleanUpDirectory(System.Object)
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)
Also this error appears in the event log as well:
EventType clr20r3, P1 myServiceName.exe, P2 1.0.1.0, P3 4db6e85d, P4 mscorlib, P5 4.0.0.0, P6 4d53693b, P7 3dab, P8 23b, P9 system.io.ioexception, P10 NIL.
Here is the code that is supposed to handle the exception, wait and retry to delete the files.
while (!isDone)
{
bool myRetVal = true;
string myErrorMsg = String.Empty;
string myFile = String.Empty;
//give it time to finish processing
Thread.Sleep(30 * 1000);
//
//delete Files in folder
foreach (string file in Directory.GetFiles(myDirectory, "*.PDF"))
{ //delete all the *.PDF files
try
{
File.Delete(file);
}
catch (Exception ex)
{
myErrorMsg = "...\r\n" + ex.Message;
m_Log.Writeline("Unable to delete "+ file + ". MESSAGE:"+ myErrorMsg,3);
myFile = file;
myRetVal = false;
}
}
//
//now move the in-progress File to the processed directory
if (myRetVal)
{
foreach (string file in Directory.GetFiles(myDirectory, "*.InProgress"))
{
try
{
if (File.Exists(m_ConfigFile.ProcessedFolderName + Path.GetFileNameWithoutExtension(file) + ".done"))
{
File.Delete(file);
}
else
{
File.Move(file, m_ConfigFile.ProcessedFolderName + Path.GetFileNameWithoutExtension(file) + ".done");
}
}
catch (Exception ex)
{
if (ex.Message.Contains("file already exists"))
{
}
else
{
myErrorMsg = "...\r\n" + ex.Message;
myFile = file;
myRetVal = false;
}
}
}
}
//
//if empty, delete the SendMailFolder subfolder
if (myRetVal)
{
try
{
if (Directory.GetFiles(myDirectory, "*.*").Length == 0) Directory.Delete(myDirectory);
}
catch (Exception ex)
{
myErrorMsg = "...\r\n" + ex.Message;
myFile = myDirectory;
myRetVal = false;
}
}
if (Console.CursorLeft > 0) Console.WriteLine("\r\n");
if (myRetVal)
{
Console.WriteLine(DateTime.Now.ToLocalTime() + " CleanUp SendMailFolder...Done");
isDone = true;
}
else
{
if (myErrorMsg.Contains("is being used by another process."))
{
myErrorMsg = " is still in use.";
Console.WriteLine(DateTime.Now.ToLocalTime() + " CleanUp SendMailFolder..." + Path.GetFileName(myFile) + myErrorMsg);
}
else
{
Console.WriteLine(DateTime.Now.ToLocalTime() + " CleanUp SendMailFolder..." + Path.GetFileName(myFile) + myErrorMsg);
m_Log.Writeline(DateTime.Now.ToLocalTime() + " CleanUp SendMailFolder..." + Path.GetFileName(myFile) + myErrorMsg, 3);
isDone = true;
}
}
}

You're trying to use the Console class in a service, which doesn't have a console window associated with it. You should use some alternate form of logging that doesn't assume there's a console window. log4net, as one example, allows you to configure multiple "appenders", such as a console, file, and event log appender, to use simultaneously (and which will simply be ignored if they aren't appropriate).
EDIT to clarify:
You can create a console window manually with the AllocConsole P/Invoke call if you absolutely need to. By default, a service doesn't interact with your desktop, so creating a console window would be a bad idea. For that to work you would need to configure the service with the "Allow service to interact with the desktop" setting turned on. This has security implications, particularly for machines that have multiple users, so I would advise against it.

It looks like there error is happening before your try catch. Given that you should see the specific message output by your logging (I assume it goes to the Event Log). I don't see a message that looks like what is in your log. You might want to have an app domain exception handler to catch any exceptions and log them.

Also you should ensure your m_log routine has proper error handling as that could be the culprit as well.

MessageQueue.BeginReceive() null ref error - c#

Have a windows service that listens to a msmq. In the OnStart method is have this
protected override void OnStart(string[] args)
{
try
{
_queue = new MessageQueue(_qPath);//this part works as i had logging before and afer this call
//Add MSMQ Event
_queue.ReceiveCompleted += new ReceiveCompletedEventHandler(queue_ReceiveCompleted);//this part works as i had logging before and afer this call
_queue.BeginReceive();//This is where it is failing - get a null reference exception
}
catch(Exception ex)
{
EventLogger.LogEvent(EventSource, EventLogType, "OnStart" + _lineFeed +
ex.InnerException.ToString() + _lineFeed + ex.Message.ToString());
}
}
where
private MessageQueue _queue = null;
This works on my machine but when deployed to a windows 2003 server and running as Network service account, it fails
Exception recvd:
Service cannot be started. System.NullReferenceException: Object reference not set to an instance of an object.
at MYService.Service.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Solved:
Turned out that the Q that i set up, I had to explicitly add Network Service account to it under security tab

You're seeing that particular exception because you're calling ex.InnerException.ToString(). The InnerException property is not always populated (in fact, it frequently isn't, nor should it be).
Your root problem is likely that the Network Service account doesn't have permissions to access the queue (in this case, read from it).
Here's some code that will help you get the actual error in your event log:
catch(Exception ex)
{
Exception e = ex;
StringBuilder message = new StringBuilder();
while(e != null)
{
if(message.Length > 0) message.AppendLine("\nInnerException:");
message.AppendLine(e.ToString());
e = e.InnerException;
}
EventLogger.LogEvent(EventSource, EventLogType, "OnStart" + _lineFeed +
message.ToString());
}