We Keep Coding

C# error propagating up the stack when it should not

I'm encountering an issue where a service is exiting on errors that should never propagate up.
I built a microservice manager (.NET as the local environment doesnt support .NET Core and some of its native microservice abilities)
Built in VS2019 targeting .NET 4.5.2 (I know, but this is the world we live in)
For the microservice manager, it is built and installed as a windows service. Entry looks like this (#if/#else was for testing locally, it is working as intended when registered as a windows service)
Program.cs (Entry point)
` static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
#if DEBUG
Scheduler myScheduler = new Scheduler();
myScheduler.OnDebug();
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
#else
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Scheduler()
};
ServiceBase.Run(ServicesToRun);
#endif
}
}`
Scheduler.cs
//(confidential code hidden)
`private static readonly Configuration config = Newtonsoft.Json.JsonConvert.DeserializeObject<Configuration>(
File.ReadAllText(configFilePath)
);
public Scheduler()
{
//InitializeComponent(); //windows service, doesnt need UI components initialized
}
public void OnDebug()
{
OnStart(null); //triggers when developing locally
}
protected override async void OnStart(string[] args)
{
try
{
logger.Log($#"Service manager starting...");
logger.Log($#"Finding external services... {config.services.Count} services found.");
foreach (var service in config.services)
{
try
{
if (service.disabled)
{
logger.Log(
$#"Skipping {service.name}: disabled=true in Data Transport Service's appSettings.json file");
continue;
}
logger.Queue($#"Starting: {service.name}...");
string serviceLocation = service.useRelativePath
? Path.Combine(assemblyLocation, service.path)
: service.path;
var svc = Assembly.LoadFrom(serviceLocation);
var assemblyType = svc.GetType($#"{svc.GetName().Name}.Program");
var methodInfo = assemblyType.GetMethod("Main");
var instanceObject = Activator.CreateInstance(assemblyType, new object[0]);
methodInfo.Invoke(instanceObject, new object[0]);
logger.Queue(" Running").Send("");
}
catch (TargetInvocationException ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
catch (Exception ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
}
logger.Log("Finished loading services.");
}
catch (Exception ex)
{
logger.Log($#"Critical error encountered", LOG.LEVEL.CRITICAL, ex);
}
}
Microservice:
public [Confidential]()
{
if (currentProfile == null)
{
var errMsg =
$#"Service not loaded, Profile not found, check appSettings.currentProfile: '{config.currentProfile}'";
logger.Log(errMsg,severity: LOG.LEVEL.CRITICAL);
throw new SettingsPropertyNotFoundException(errMsg);
}
if (currentProfile.disabled)
{
var errMsg = $#"Service not loaded: {config.serviceName}, Service's appSettings.currentProfile.disabled=true";
logger.Log(errMsg,LOG.LEVEL.WARN);
throw new ArgumentException(errMsg);
}
logger.Log($#"Loading: '{config.serviceName}' with following configuration:{Environment.NewLine}{JsonConvert.SerializeObject(currentProfile,Formatting.Indented)}");
logger.Queue($#"Encrypting config file passwords...");
bool updateConfig = false;
foreach (var kafkaSource in config.dataTargets)
{
if (!kafkaSource.password.IsEncrypted())
{
updateConfig = true;
logger.Queue($#"%tabEncrypting: {kafkaSource.name}");
kafkaSource.password = kafkaSource.password.Encrypt();
}
else
{
logger.Queue($#"%tabAlready encrypted: {kafkaSource.name}");
}
}
logger.Send(Environment.NewLine);
if (updateConfig)
{
File.WriteAllText(
configFilePath,
Newtonsoft.Json.JsonConvert.SerializeObject(config));
}
var _source = config.dataSources.FirstOrDefault(x=>x.name==currentProfile.dataSource);
var _target = config.dataTargets.FirstOrDefault(x => x.name == currentProfile.dataTarget);
source = new Connectors.Sql(logger,
_source?.name,
_source?.connectionString,
_source.pollingInterval,
_source.maxRowsPerSelect,
_source.maxRowsPerUpdate);
target = new Connectors.KafkaProducer(logger)
{
bootstrapServers = _target?.bootstrapServers,
name = _target?.name,
password = _target?.password.Decrypt(),
sslCaLocation = Path.Combine(assemblyLocation,_target?.sslCaLocation),
topic = _target?.topic,
username = _target?.username
};
Start();
}
public void Start()
{
Timer timer = new Timer();
try
{
logger.Log($#"SQL polling interval: {source.pollingInterval} seconds");
timer.Interval = source.pollingInterval * 1000;
timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
timer.Start();
if (currentProfile.executeOnStartup)
Run();
}
catch (Exception ex)
{
var sb = new StringBuilder();
sb.AppendLine($#"Critical error encountered loading external service: {config.serviceName}.");
if (!timer.Enabled)
sb.AppendLine($#"service unloaded - Schedule not started!");
else
sb.AppendLine($#"service appears to be loaded and running on schedule.");
logger.Log(sb.ToString(), LOG.LEVEL.CRITICAL, ex);
}
}
public void OnTimer(object sender, ElapsedEventArgs e)
{
try
{
Run();
}
catch (Exception ex)
{
logger.Log($#"Critical error during scheduled run on service: {config.serviceName}.", LOG.LEVEL.CRITICAL, ex);
}
}
public async void Run()
{
//Get new alarm events from SQL source
logger.Queue("Looking for new alarms...");
var rows = await GetNewEvents();`
The exception occurred during the GetNewEvents method, which attempted to open a SqlConnection to a SQL server that was unavailable due to network issues, that method intentionally throws an exception, which should throw up to OnTimer, where it gets caught, logged, and the timer keeps running. During development/testing, I used invalid credentials, bad connection string, etc and simulated this type of error and it worked as expected, logged the error, kept running. For some reason recently, that error is not caught in OnTimer, it propagates up, where it should be caught by Start (but isn't), after that it should be caught by the parent service manager which is entirely wrapped in a try/catch with no throw's, and above that (because their could be multiple microservices managed by that service) the entry point to the service manager is wrapped in try/catch with no throws, all for isolation from microservice errors. For some reason though, now, the error from a VERY downstream application is propagating all the way up.
Typically, this code runs 24/7 no issues, the microservice it is loading from the config file launches and runs fine. The entry into that specific microservice starts with a try {...} catch (Exception ex) {...} block.
The concept is to have a microservice manager than can launch a number of microservices without having to install all of them as windows services, and have some level of configuration driven by a config file that dictates how the main service runs.
The microservice represented here opens a SQL connection, reads data, performs business logic, publishes results to Kafka, it does this on a polling interval dictated by the config file contained in the microservice. As stated above, its ran for months without issue.
Recently, I noticed the main microservice manager service was not running on the windows server, I investigated the Server Application Logs and found a "Runtime Error" that essentially stated the microservice, while attempting to connect to sql, failed (network issue) and caused the entire microservice manager to exit. To my understanding, they way I'm launching the microservice should isolate it from the main service manager app. Additionally, the main service manager app is wrapped in a very generic try catch block. The entry point to the micro service itself is wrapped in a try catch, and almost every component in the microservice is wrapped in try / catch per business need. The scenario that faulted (cant connect to sql) intentionally throws an error for logging purposes, but should be caught by the immediate parent try/catch, which does not propagate or re-throw, only logs the error to a txt file and the windows server app log.
How is it that this exception is bubbling up through isolation points and causing the main service to fault and exit? I tested this extensively during development and prior to release, this exact scenario being unable to connect to sql, and it generated the correct log entry, and tried again on the next polling cycle as expected.
I haven't tried any other approaches as yet, as I feel they would be band-aid fixes as best as I dont understand why the original design is suddenly failing. The server hasn't changed, no patching/security updates/etc.
From the server Application Log:
Application: DataTransportService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
at Connectors.SqlHelper.DbHelper+d__13`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__14.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

If you throw/generate an exception from an asynchronous callback where does/can/should it be caught?

Say we're using Socket.BeginReceive and passing in a callback method:
var state = new StateObject() { workSocket = socket };
var ret = socket.BeginReceive(state.buffer, 0, StateObject.BufferSize, SocketFlags.None, new AsyncCallback(BeginReceiveCallback), state);
...
private void BeginReceiveCallback(IAsyncResult ar)
{
StateObject state = (StateObject) ar.AsyncState;
int bytes = state.workSocket.EndReceive(ar); //workSocket is the Socket receiving
...
}
If an exception is encountered in the callback and not handled there - where will it go? How/can I catch it?

Doing a bit of reading, I believe this is the answer
The Exceptions get thrown when you call EndReceive. The Begin.../End... pair of methods work like this:
Begin gets called, and returns immediately
The callback gets started in a separate Thread by the runtime
Inside the callback, the actual work gets done blocking the Thread. This work is done by invoking the End... method
So, the End... method is actually doing the work. So, if an exception gets thrown, you can catch it there.
Here is some code demonstrating this with comments. If you want to just try it out here is a Dotnetfiddle with this code edited to work on Dotnetfiddle:
using System;
using System.Threading;
using System.Threading.Tasks;
namespace BeginEndInvokeTest
{
public class Program
{
public static async Task Main(string[] args)
{
// This is just here for setup
var caller = new AsyncDemo.AsyncMethodCaller(AsyncDemo.Test);
// This is your 'socket.BeginReceive' call
var ar = caller.BeginInvoke(3000, AsyncCallback, caller);
// Wait so the program doesn't exit prematurely
await Task.Delay(5000);
}
static void AsyncCallback(IAsyncResult ar)
{
var caller = (AsyncDemo.AsyncMethodCaller) ar.AsyncState;
try
{
// If our exception wouldn't be thrown here (which is impossible),
// the program would print "No exception was thrown"
caller.EndInvoke(ar);
Console.WriteLine("No exception was thrown");
}
catch (Exception)
{
Console.WriteLine("Exception encountered");
}
}
}
public class AsyncDemo
{
public static string Test(int callDuration)
{
// Simply write something to the console, simulate work
// and throw an exception
Console.WriteLine("Test method begins");
Thread.Sleep(callDuration);
throw new Exception("Testing");
}
public delegate string AsyncMethodCaller(int callDuration);
}
}
So in short, you can only catch the Exceptions at the End... call, nowhere else.
Edit to address where does the exception go when it isn't caught.
Honestly, I have no idea where it goes. Further testing and trial n' error gave me nothing. It seems like the whole runtime just crashes. When I didn't catch the exception I get a console out with a stack trace that shows where the exception was thrown (inside the Test method, as expected), alongside something I've never seen before: Unhandled Exception: System.Exception: Testing.
There is also a second stack trace saying :
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
...
So, yeah, it seems like the runtime crashes when you don't catch it.
Source: I cobbled this answer together using this Microsoft API documentation. Some further info can be found here as Calling synchronous methods asynchronously.

Unhandled DivideByZero exception from an external DLL - C#

I have a C# (.net 4.0) program, whose main is calling methods from an external FTP library - a dll the project references. The logic is in a try-catch block, and the catch prints the error. The exception handler has a generic parameter: catch(Exception ex). The IDE is VS.
Sometimes the FTP library throws the following division by zero exception. The problem is it is not caught in the catch block, and the program crashes.
Exceptions originated in my wrapper code are caught. Anyone has any idea what the difference is and how the exception can be caught?
The exception:
Description: The process was terminated due to an unhandled exception.
Exception Info: System.DivideByZeroException
Stack:
at ComponentPro.IO.FileSystem+c_OU.c_F2B()
at System.Threading.ExecutionContext.runTryCode(System.Object)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

There is a similar problem described here and also here for explanation. As said in one of the comments an FTP server should always handle protocol violations itself without crashing. You should pick another FTP if you can. However, if you want to keep using that DLL you need to handle the exception at App Domain level as Blorgbeard pointed out.
Here an example of how catch the exception using the AppDomain.UnhandledException event:
using System;
using System.Security.Permissions;
public class Test
{
[SecurityPermission(SecurityAction.Demand, Flags = SecurityPermissionFlag.ControlAppDomain)]
public static void Example()
{
AppDomain currentDomain = AppDomain.CurrentDomain;
currentDomain.UnhandledException += new UnhandledExceptionEventHandler(MyHandler);
try
{
throw new Exception("1");
}
catch (Exception e)
{
Console.WriteLine("Catch clause caught : " + e.Message);
}
throw new Exception("2");
// Output:
// Catch clause caught : 1
// MyHandler caught : 2
}
static void MyHandler(object sender, UnhandledExceptionEventArgs args)
{
Exception e = (Exception)args.ExceptionObject;
Console.WriteLine("MyHandler caught : " + e.Message);
}
public static void Main()
{
Example();
}
}

Cleanly interrupt HttpListener's BeginGetContext method

I am using a HttpListener and using BeginGetContext to get my context object. I want to cleanly shut down my HttpListener but if I attempt to do a Close on the listener I get a exception and it causes my program to exit.
using System;
using System.Net;
namespace Sandbox_Console
{
class Program
{
public static void Main()
{
if (!HttpListener.IsSupported)
{
Console.WriteLine("Windows XP SP2 or Server 2003 is required to use the HttpListener class.");
return;
}
// Create a listener.
HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://vwdev.vw.local:8081/BitsTest/");
listener.Start();
Console.WriteLine("Listening...");
listener.BeginGetContext(Context, listener);
Console.ReadLine();
listener.Close(); //Exception on this line, but not shown in Visual Studio
Console.WriteLine("Stopped Listening..."); //This line is never reached.
Console.ReadLine();
}
static void Context(IAsyncResult result)
{
HttpListener listener = (HttpListener)result.AsyncState;
HttpListenerContext context = listener.EndGetContext(result);
context.Response.Close();
listener.BeginGetContext(Context, listener);
}
}
}
The program throws a exception on the listener.Close() however the error never gets shown in Visual Studio, the only note I get is the following in the Debug output screen:
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
The program '[2568] Sandbox Console.vshost.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).
I was able to get the real execption from windows Event Viewer
Application: Sandbox Console.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException
Stack:
at System.Net.HttpListener.EndGetContext(System.IAsyncResult)
at Sandbox_Console.Program.Context(System.IAsyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr)
at System.Net.ListenerAsyncResult.WaitCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
What do I need to do so I can close my HttpListener cleanly?

Context gets called one last time when you call Close, you must handle the object disposed exception that could get thrown
static void Context(IAsyncResult result)
{
HttpListener listener = (HttpListener)result.AsyncState;
try
{
//If we are not listening this line throws a ObjectDisposedException.
HttpListenerContext context = listener.EndGetContext(result);
context.Response.Close();
listener.BeginGetContext(Context, listener);
}
catch (ObjectDisposedException)
{
//Intentionally not doing anything with the exception.
}
}

You could add this line
if (!listener.IsListening) { return; }
HttpListenerContext context = listener.EndGetContext(ctx);

NamedPipes throwing an exception in windows service

I have created a windows service which allows communications via namedpipes.
This code worked fine when I wrote some unit tests to spin up the pipes and test the communication, but now I have installed the same code in my windows service I get the following error:
Exception Info: System.IO.IOException
Stack:
at System.IO.__Error.WinIOError(Int32, System.String)
at System.IO.Pipes.NamedPipeServerStream.Create(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode, System.IO.Pipes.PipeOptions, Int32, Int32, System.IO.Pipes.PipeAccessRights, SECURITY_ATTRIBUTES)
at System.IO.Pipes.NamedPipeServerStream..ctor(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode, System.IO.Pipes.PipeOptions, Int32, Int32, System.IO.Pipes.PipeSecurity, System.IO.HandleInheritability, System.IO.Pipes.PipeAccessRights)
at System.IO.Pipes.NamedPipeServerStream..ctor(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode,
System.IO.Pipes.PipeOptions, Int32, Int32, System.IO.Pipes.PipeSecurity)
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)
Now I did some googling and found this post in stackoverflow > POST But I implemented this (apart from the ps.AddAccessRule(pa); as that made no reference to was pa was)
and I get the same error.
this is the code I have for the thread:
var pipeSecurity = new PipeSecurity();
pipeSecurity.AddAccessRule(new PipeAccessRule("Users", PipeAccessRights.ReadWrite | PipeAccessRights.CreateNewInstance, AccessControlType.Allow));
pipeSecurity.AddAccessRule(new PipeAccessRule("CREATOR OWNER", PipeAccessRights.FullControl, AccessControlType.Allow));
pipeSecurity.AddAccessRule(new PipeAccessRule("SYSTEM", PipeAccessRights.FullControl, AccessControlType.Allow));
var pipeServer = new NamedPipeServerStream(pipeName, PipeDirection.InOut, numThreads, PipeTransmissionMode.Message, PipeOptions.Asynchronous, 1024, 1024, pipeSecurity);
pipeServer.WaitForConnection();
any help will be great.
Ok here is the code that is running the listener:
the windows service:
public static System.Timers.Timer Timer = new System.Timers.Timer();
public void Start()
{
Timer.Elapsed += (HeartBeat);
//Timer.Interval = 100; //Live
Timer.Interval = 2000; //Debug
Timer.Start();
}
public void Stop()
{
Timer.Stop();
}
private static void HeartBeat(object sender, ElapsedEventArgs e)
{
//listen for a message
ListenForMessage();
}
the listener code:
private const String pipeName = "StackOVerFlowPipeCode";
private const int numThreads = 10;
public static void ListenForMessage()
{
int i;
var servers = new Thread[numThreads];
for (i = 0; i < numThreads; i++)
{
servers[i] = new Thread(ServerThread);
servers[i].Start();
}
Thread.Sleep(250);
while (i > 0)
{
for (var j = 0; j < numThreads; j++)
{
if (servers[j] == null) continue;
if (!servers[j].Join(250)) continue;
servers[j] = null;
i--; // decrement the thread watch count
}
}
}
private static void ServerThread(object data)
{
try
{
var pipeServer = new NamedPipeServerStream(pipeName, PipeDirection.InOut, numThreads);
pipeServer.WaitForConnection();
var ss = new StreamString(pipeServer);
ss.WriteString(pipeName);
var message = ss.ReadString();
//DO STUFF HERE WITH MESSAGE
pipeServer.Close();
}
catch (Exception ex)
{
//CRY LIKE A BABY WHO LOST HIS TEDDY
throw ex;
}
}
Exception message found: All pipe instances are busy.

Turns out my whole implementation was flawed, I deleted a lot of code and rewrote it and it worked. I removed code from in ListenForMessage() and replaced it with the code from ServerThread() and then also changed how the service was calling this, to a thread from a timer. and it worked.
well sort of, the code after the message is received (shown as //Do Stuff in the above) does work, but at least this task is complete.
note: never just copy and past code and take it for granted always read it and understand it.

Named Pipe names need to be in the format:
\\.\pipe\pipename
See CreatedNamedPipe Win32 API: http://msdn.microsoft.com/en-us/library/aa365150(VS.85).aspx
EDIT: On further examination (looking at the NamedPipeServerStream sample) this might not be the case. So need to see the full details of the exception: Message property and run time type.

You didn't include the exception.Message but According to the MSDN page for NamedPipeServerStream, the IOException is for "The maximum number of server instances has been exceeded." is this the exception you are getting?
If so, what is numThreads set to, and is it possible you have another instance of you app running? (e.g. the unit test you were experimenting with?)