We Keep Coding

C# error propagating up the stack when it should not

I'm encountering an issue where a service is exiting on errors that should never propagate up.
I built a microservice manager (.NET as the local environment doesnt support .NET Core and some of its native microservice abilities)
Built in VS2019 targeting .NET 4.5.2 (I know, but this is the world we live in)
For the microservice manager, it is built and installed as a windows service. Entry looks like this (#if/#else was for testing locally, it is working as intended when registered as a windows service)
Program.cs (Entry point)
` static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
#if DEBUG
Scheduler myScheduler = new Scheduler();
myScheduler.OnDebug();
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
#else
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Scheduler()
};
ServiceBase.Run(ServicesToRun);
#endif
}
}`
Scheduler.cs
//(confidential code hidden)
`private static readonly Configuration config = Newtonsoft.Json.JsonConvert.DeserializeObject<Configuration>(
File.ReadAllText(configFilePath)
);
public Scheduler()
{
//InitializeComponent(); //windows service, doesnt need UI components initialized
}
public void OnDebug()
{
OnStart(null); //triggers when developing locally
}
protected override async void OnStart(string[] args)
{
try
{
logger.Log($#"Service manager starting...");
logger.Log($#"Finding external services... {config.services.Count} services found.");
foreach (var service in config.services)
{
try
{
if (service.disabled)
{
logger.Log(
$#"Skipping {service.name}: disabled=true in Data Transport Service's appSettings.json file");
continue;
}
logger.Queue($#"Starting: {service.name}...");
string serviceLocation = service.useRelativePath
? Path.Combine(assemblyLocation, service.path)
: service.path;
var svc = Assembly.LoadFrom(serviceLocation);
var assemblyType = svc.GetType($#"{svc.GetName().Name}.Program");
var methodInfo = assemblyType.GetMethod("Main");
var instanceObject = Activator.CreateInstance(assemblyType, new object[0]);
methodInfo.Invoke(instanceObject, new object[0]);
logger.Queue(" Running").Send("");
}
catch (TargetInvocationException ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
catch (Exception ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
}
logger.Log("Finished loading services.");
}
catch (Exception ex)
{
logger.Log($#"Critical error encountered", LOG.LEVEL.CRITICAL, ex);
}
}
Microservice:
public [Confidential]()
{
if (currentProfile == null)
{
var errMsg =
$#"Service not loaded, Profile not found, check appSettings.currentProfile: '{config.currentProfile}'";
logger.Log(errMsg,severity: LOG.LEVEL.CRITICAL);
throw new SettingsPropertyNotFoundException(errMsg);
}
if (currentProfile.disabled)
{
var errMsg = $#"Service not loaded: {config.serviceName}, Service's appSettings.currentProfile.disabled=true";
logger.Log(errMsg,LOG.LEVEL.WARN);
throw new ArgumentException(errMsg);
}
logger.Log($#"Loading: '{config.serviceName}' with following configuration:{Environment.NewLine}{JsonConvert.SerializeObject(currentProfile,Formatting.Indented)}");
logger.Queue($#"Encrypting config file passwords...");
bool updateConfig = false;
foreach (var kafkaSource in config.dataTargets)
{
if (!kafkaSource.password.IsEncrypted())
{
updateConfig = true;
logger.Queue($#"%tabEncrypting: {kafkaSource.name}");
kafkaSource.password = kafkaSource.password.Encrypt();
}
else
{
logger.Queue($#"%tabAlready encrypted: {kafkaSource.name}");
}
}
logger.Send(Environment.NewLine);
if (updateConfig)
{
File.WriteAllText(
configFilePath,
Newtonsoft.Json.JsonConvert.SerializeObject(config));
}
var _source = config.dataSources.FirstOrDefault(x=>x.name==currentProfile.dataSource);
var _target = config.dataTargets.FirstOrDefault(x => x.name == currentProfile.dataTarget);
source = new Connectors.Sql(logger,
_source?.name,
_source?.connectionString,
_source.pollingInterval,
_source.maxRowsPerSelect,
_source.maxRowsPerUpdate);
target = new Connectors.KafkaProducer(logger)
{
bootstrapServers = _target?.bootstrapServers,
name = _target?.name,
password = _target?.password.Decrypt(),
sslCaLocation = Path.Combine(assemblyLocation,_target?.sslCaLocation),
topic = _target?.topic,
username = _target?.username
};
Start();
}
public void Start()
{
Timer timer = new Timer();
try
{
logger.Log($#"SQL polling interval: {source.pollingInterval} seconds");
timer.Interval = source.pollingInterval * 1000;
timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
timer.Start();
if (currentProfile.executeOnStartup)
Run();
}
catch (Exception ex)
{
var sb = new StringBuilder();
sb.AppendLine($#"Critical error encountered loading external service: {config.serviceName}.");
if (!timer.Enabled)
sb.AppendLine($#"service unloaded - Schedule not started!");
else
sb.AppendLine($#"service appears to be loaded and running on schedule.");
logger.Log(sb.ToString(), LOG.LEVEL.CRITICAL, ex);
}
}
public void OnTimer(object sender, ElapsedEventArgs e)
{
try
{
Run();
}
catch (Exception ex)
{
logger.Log($#"Critical error during scheduled run on service: {config.serviceName}.", LOG.LEVEL.CRITICAL, ex);
}
}
public async void Run()
{
//Get new alarm events from SQL source
logger.Queue("Looking for new alarms...");
var rows = await GetNewEvents();`
The exception occurred during the GetNewEvents method, which attempted to open a SqlConnection to a SQL server that was unavailable due to network issues, that method intentionally throws an exception, which should throw up to OnTimer, where it gets caught, logged, and the timer keeps running. During development/testing, I used invalid credentials, bad connection string, etc and simulated this type of error and it worked as expected, logged the error, kept running. For some reason recently, that error is not caught in OnTimer, it propagates up, where it should be caught by Start (but isn't), after that it should be caught by the parent service manager which is entirely wrapped in a try/catch with no throw's, and above that (because their could be multiple microservices managed by that service) the entry point to the service manager is wrapped in try/catch with no throws, all for isolation from microservice errors. For some reason though, now, the error from a VERY downstream application is propagating all the way up.
Typically, this code runs 24/7 no issues, the microservice it is loading from the config file launches and runs fine. The entry into that specific microservice starts with a try {...} catch (Exception ex) {...} block.
The concept is to have a microservice manager than can launch a number of microservices without having to install all of them as windows services, and have some level of configuration driven by a config file that dictates how the main service runs.
The microservice represented here opens a SQL connection, reads data, performs business logic, publishes results to Kafka, it does this on a polling interval dictated by the config file contained in the microservice. As stated above, its ran for months without issue.
Recently, I noticed the main microservice manager service was not running on the windows server, I investigated the Server Application Logs and found a "Runtime Error" that essentially stated the microservice, while attempting to connect to sql, failed (network issue) and caused the entire microservice manager to exit. To my understanding, they way I'm launching the microservice should isolate it from the main service manager app. Additionally, the main service manager app is wrapped in a very generic try catch block. The entry point to the micro service itself is wrapped in a try catch, and almost every component in the microservice is wrapped in try / catch per business need. The scenario that faulted (cant connect to sql) intentionally throws an error for logging purposes, but should be caught by the immediate parent try/catch, which does not propagate or re-throw, only logs the error to a txt file and the windows server app log.
How is it that this exception is bubbling up through isolation points and causing the main service to fault and exit? I tested this extensively during development and prior to release, this exact scenario being unable to connect to sql, and it generated the correct log entry, and tried again on the next polling cycle as expected.
I haven't tried any other approaches as yet, as I feel they would be band-aid fixes as best as I dont understand why the original design is suddenly failing. The server hasn't changed, no patching/security updates/etc.
From the server Application Log:
Application: DataTransportService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
at Connectors.SqlHelper.DbHelper+d__13`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__14.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

If you throw/generate an exception from an asynchronous callback where does/can/should it be caught?

Say we're using Socket.BeginReceive and passing in a callback method:
var state = new StateObject() { workSocket = socket };
var ret = socket.BeginReceive(state.buffer, 0, StateObject.BufferSize, SocketFlags.None, new AsyncCallback(BeginReceiveCallback), state);
...
private void BeginReceiveCallback(IAsyncResult ar)
{
StateObject state = (StateObject) ar.AsyncState;
int bytes = state.workSocket.EndReceive(ar); //workSocket is the Socket receiving
...
}
If an exception is encountered in the callback and not handled there - where will it go? How/can I catch it?

Doing a bit of reading, I believe this is the answer
The Exceptions get thrown when you call EndReceive. The Begin.../End... pair of methods work like this:
Begin gets called, and returns immediately
The callback gets started in a separate Thread by the runtime
Inside the callback, the actual work gets done blocking the Thread. This work is done by invoking the End... method
So, the End... method is actually doing the work. So, if an exception gets thrown, you can catch it there.
Here is some code demonstrating this with comments. If you want to just try it out here is a Dotnetfiddle with this code edited to work on Dotnetfiddle:
using System;
using System.Threading;
using System.Threading.Tasks;
namespace BeginEndInvokeTest
{
public class Program
{
public static async Task Main(string[] args)
{
// This is just here for setup
var caller = new AsyncDemo.AsyncMethodCaller(AsyncDemo.Test);
// This is your 'socket.BeginReceive' call
var ar = caller.BeginInvoke(3000, AsyncCallback, caller);
// Wait so the program doesn't exit prematurely
await Task.Delay(5000);
}
static void AsyncCallback(IAsyncResult ar)
{
var caller = (AsyncDemo.AsyncMethodCaller) ar.AsyncState;
try
{
// If our exception wouldn't be thrown here (which is impossible),
// the program would print "No exception was thrown"
caller.EndInvoke(ar);
Console.WriteLine("No exception was thrown");
}
catch (Exception)
{
Console.WriteLine("Exception encountered");
}
}
}
public class AsyncDemo
{
public static string Test(int callDuration)
{
// Simply write something to the console, simulate work
// and throw an exception
Console.WriteLine("Test method begins");
Thread.Sleep(callDuration);
throw new Exception("Testing");
}
public delegate string AsyncMethodCaller(int callDuration);
}
}
So in short, you can only catch the Exceptions at the End... call, nowhere else.
Edit to address where does the exception go when it isn't caught.
Honestly, I have no idea where it goes. Further testing and trial n' error gave me nothing. It seems like the whole runtime just crashes. When I didn't catch the exception I get a console out with a stack trace that shows where the exception was thrown (inside the Test method, as expected), alongside something I've never seen before: Unhandled Exception: System.Exception: Testing.
There is also a second stack trace saying :
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
...
So, yeah, it seems like the runtime crashes when you don't catch it.
Source: I cobbled this answer together using this Microsoft API documentation. Some further info can be found here as Calling synchronous methods asynchronously.

Running TCPListener from a Windows Service

I am trying to create a Windows service which listens on a particular port for an incoming serialized object and then performs some action, however every time I start I get the following error:
System.Net.Sockets.SocketException (0x80004005): Only one usage of each socket address (protocol/network address/port) is normally permitted
Here is the code that's running:
protected override void OnStart(string[] args)
{
Thread t = new Thread(Server.StartServer);
t.Start();
}
The StartServer method looks like this:
public static void StartServer()
{
TcpListener server = null;
try
{
server = new TcpListener(IPAddress.Loopback, 8889);
server.Start();
}
catch(Exception e)
{
//write to a log file and close
}
for (; ; )
//Some more code that handles the serialization etc
The message seems to state that something else is already listening on that socket however when I take a look using CMD and netstat -a I see nothing on that port, and it seems to occur regardless of what I change the listening port to. The exception occurs on the line "server = new TcpListener(IPAddress.Loopback, 8889);".
Why does this error occur and how am I able to make this work properly?
*Edited to note that I have also tried this as a console app and it seems to work, it only fails when running as a Windows Service

C# Doing retry on tcp socket communication

I'm trying to do a retry coding if my client fail to connect to my server. Below is how i do:
In main function:
Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
client.BeginConnect(remoteEP, new AsyncCallback(ConnectCallback), client);
ConnectCallback:
private void ConnectCallback(IAsyncResult ar)
{
try
{
// Retrieve the socket from the state object.
Socket client = (Socket)ar.AsyncState;
// Complete the connection.
client.EndConnect(ar);
_logger.Info("## Connection to server successful at " + strServerIP + ":" + strServerPort);
}
catch (Exception ex)
{
_logger.Info("## Connection to server failed. Retrying...");
Socket client = (Socket)ar.AsyncState;
IPEndPoint remoteEP = new IPEndPoint(IPAddress.Parse(strServerIP), Convert.ToInt32(strServerPort));
client.BeginConnect(remoteEP, new AsyncCallback(ConnectCallback), client);
}
}
I will catch the exception in ConnectCallback when connection fail and do retry.
But I found out that, if it retried 10 times, when server is up, the server will get 10 connection from that same client. If it retry 50 times, when server is up, then server will get 50 connections.
Is my coding something wrong? Seems like everytime it retry, my server will get a new connection.

Without a functioning example it's difficult to know. If this is close to what you are actually doing, I suspect a couple of things are wrong. The Socket object appears to be blocking by default, but something is generating your exception and it may not be what you think it is. First thing to do is only catch the SocketException and then only retry when the exception represents something that might indicate a retry would work. Put in a delay because if it didn't work 1 ms ago, it probably won't work now. Put in a sanity counter so it gives up retrying after so many tries. Examine your protocol to make sure you are sending the server what it expects. Above all Close your sockets.
What I suspect you are seeing is a bunch of Socket connections caused by an exception (which may or may not be related to the sockets). Since you never close them they just build up. I suspect that eventually the GC might kick in and run finalizers on the objects which would then drop their connections. More likely the server would drop the connections. Either way if you aren't explicitly closing your Socket then it will hang around until something times out.
Here is a working example that demonstrates what I think you are asking. Again you need to decide under what conditions you should retry because retrying if anything goes wrong is not good. It may cause your program to constantly churn threads and possibly even connections.
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using Microsoft.Extensions.Logging;
namespace socketTst {
class Program {
static ILoggerFactory loggerFactory = new LoggerFactory().AddConsole().AddDebug();
static ILogger _logger;
static AutoResetEvent finish = new AutoResetEvent(false);
static String Hostname = "www.google.com";
static int Port = 80;
static int RetryCount = 0;
static void ConnectCallback(IAsyncResult ar) {
_logger.LogInformation($"## ConnectCallback entered");
// Retrieve the socket from the state object.
Socket client = (Socket) ar.AsyncState;
try {
// Complete the connection.
client.EndConnect(ar);
var s = new byte[] { 1 };
client.Send(s);
var buf = new byte[1024];
var cnt = client.Receive(buf);
_logger.LogInformation($"## Connection to server successful at {client.RemoteEndPoint}");
if (cnt > 0) {
var returned = Encoding.UTF8.GetString(buf, 0, cnt);
_logger.LogInformation($"## Data returned: {returned}");
}
else {
_logger.LogInformation($"## No data returned");
}
finish.Set(); // signal end of program
}
catch (SocketException sockExcep) {
_logger.LogInformation($"## Exception: {sockExcep.Message}");
_logger.LogInformation("## Connection to server failed. Retrying...");
// This is a bad idea. You don't know what is wrong so retrying might not be useful.
// What if this is an unknown host or some other error that isn't likely to be
// resolved by a retry ???
RetryCount++;
if (RetryCount > 10) {
_logger.LogInformation("## Not able to reach host after 10 tries");
finish.Set(); // signal end of program
return; // give up
}
Thread.Sleep(797); // wait a bit
var dest = new DnsEndPoint(Hostname, Port);
client.BeginConnect(dest, new AsyncCallback(ConnectCallback), client);
}
catch (Exception ex) {
_logger.LogInformation($"## Exception: {ex.Message}");
}
_logger.LogInformation($"## ConnectCallback exited");
}
static void Main(string[] args) {
_logger = loggerFactory.CreateLogger<Program>();
Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
client.Blocking = true;
var dest = new DnsEndPoint(Hostname, Port);
_logger.LogInformation($"Attempting connection to {dest.Host}:{dest.Port}");
_logger.LogInformation($"Socket blocking: {client.Blocking}");
_logger.LogInformation("Calling BeginConnect");
var thd = client.BeginConnect(dest, new AsyncCallback(ConnectCallback), client);
_logger.LogInformation("BeginConnect complete");
_logger.LogInformation("Calling WaitOne");
finish.WaitOne(); // don't let program end until connection is made
_logger.LogInformation("WaitOne complete");
client.Close();
Thread.Sleep(25); // if you don't do this the program ends before all the log output can be written
Console.WriteLine("Program complete");
}
}
}
I've tested this code using .NET Core 2.1 and you'll need the following nuget packages to run it:
Microsoft.Extensions.Logging
Microsoft.Extensions.Logging.Console
Microsoft.Extensions.Logging.Debug"
Successful execution looks like this:
info: socketTst.Program[0]
Attempting connection to www.google.com:80
info: socketTst.Program[0]
Socket blocking: True
info: socketTst.Program[0]
Calling BeginConnect
info: socketTst.Program[0]
BeginConnect complete
info: socketTst.Program[0]
Calling WaitOne
info: socketTst.Program[0]
## ConnectCallback entered
info: socketTst.Program[0]
## Connection to server successful at 172.217.15.68:80
info: socketTst.Program[0]
## Data returned: HTTP/1.0 400 Bad Request
Content-Length: 54
Content-Type: text/html; charset=UTF-8
Date: Wed, 26 Sep 2018 03:32:39 GMT
<html><title>Error 400 (Bad Request)!!1</title></html>
Program complete

TcpListener.AcceptSocket() fails in a Windows Service

I have an instance of the following code that executes correctly in Debug or as a standalone Windows application:
TcpListener tcpListener = new TcpListener(IPAddress.Any, 4554);
tcpListener.Start();
while (true)
{
try
{
using (Socket socket = tcpListener.AcceptSocket())
{
// Code here is reached in Debug or as a Console Application
// but not as a Windows Service
}
}
catch (SocketException se)
{
// This is never reached
}
catch (Exception ex)
{
// This is never reached
}
finally
{
// This is never reached in the Windows Service
}
}
However, when I install it as a Windows Service, it crashes on tcpListener.AcceptSocket(), and logs the following to the Event Viewer:
An unhandled exception ('System.Net.Sockets.SocketException') occurred in MyService.exe [768]. Just-In-Time debugging this exception failed with the following error: The operation attempted is not supported.
Even trying to catch the exception I am unable to log anything more. Stepping through code in Debug accomplishes nothing because the code successfully blocks the application and waits for a client connection.
Is there a way to implement this for a Windows Service?

usr's advice (and this answer) led me to a bug in the code. The ServiceBase class contained the following:
protected override void OnStart(string[] args)
{
_worker = new Thread(ExecuteService);
_worker.Start();
}
private void ExecuteService()
{
for (;;)
{
if (_stop.WaitOne(1000))
{
new TcpServer().StartTcpServer();
return;
}
}
}
The correct implementation was to remove the for loop, which was re-instantiating the listener. Here is the final code:
protected override void OnStart(string[] args)
{
_worker = new Thread(ExecuteService);
_worker.Start();
}
private static void ExecuteService()
{
new TcpServer().StartTcpServer();
}