We Keep Coding

C# error propagating up the stack when it should not

I'm encountering an issue where a service is exiting on errors that should never propagate up.
I built a microservice manager (.NET as the local environment doesnt support .NET Core and some of its native microservice abilities)
Built in VS2019 targeting .NET 4.5.2 (I know, but this is the world we live in)
For the microservice manager, it is built and installed as a windows service. Entry looks like this (#if/#else was for testing locally, it is working as intended when registered as a windows service)
Program.cs (Entry point)
` static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
#if DEBUG
Scheduler myScheduler = new Scheduler();
myScheduler.OnDebug();
System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
#else
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Scheduler()
};
ServiceBase.Run(ServicesToRun);
#endif
}
}`
Scheduler.cs
//(confidential code hidden)
`private static readonly Configuration config = Newtonsoft.Json.JsonConvert.DeserializeObject<Configuration>(
File.ReadAllText(configFilePath)
);
public Scheduler()
{
//InitializeComponent(); //windows service, doesnt need UI components initialized
}
public void OnDebug()
{
OnStart(null); //triggers when developing locally
}
protected override async void OnStart(string[] args)
{
try
{
logger.Log($#"Service manager starting...");
logger.Log($#"Finding external services... {config.services.Count} services found.");
foreach (var service in config.services)
{
try
{
if (service.disabled)
{
logger.Log(
$#"Skipping {service.name}: disabled=true in Data Transport Service's appSettings.json file");
continue;
}
logger.Queue($#"Starting: {service.name}...");
string serviceLocation = service.useRelativePath
? Path.Combine(assemblyLocation, service.path)
: service.path;
var svc = Assembly.LoadFrom(serviceLocation);
var assemblyType = svc.GetType($#"{svc.GetName().Name}.Program");
var methodInfo = assemblyType.GetMethod("Main");
var instanceObject = Activator.CreateInstance(assemblyType, new object[0]);
methodInfo.Invoke(instanceObject, new object[0]);
logger.Queue(" Running").Send("");
}
catch (TargetInvocationException ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
catch (Exception ex)
{
logger.Queue(" Failed").Send("");
logger.Log("an error occurred", LOG.LEVEL.CRITICAL, ex);
}
}
logger.Log("Finished loading services.");
}
catch (Exception ex)
{
logger.Log($#"Critical error encountered", LOG.LEVEL.CRITICAL, ex);
}
}
Microservice:
public [Confidential]()
{
if (currentProfile == null)
{
var errMsg =
$#"Service not loaded, Profile not found, check appSettings.currentProfile: '{config.currentProfile}'";
logger.Log(errMsg,severity: LOG.LEVEL.CRITICAL);
throw new SettingsPropertyNotFoundException(errMsg);
}
if (currentProfile.disabled)
{
var errMsg = $#"Service not loaded: {config.serviceName}, Service's appSettings.currentProfile.disabled=true";
logger.Log(errMsg,LOG.LEVEL.WARN);
throw new ArgumentException(errMsg);
}
logger.Log($#"Loading: '{config.serviceName}' with following configuration:{Environment.NewLine}{JsonConvert.SerializeObject(currentProfile,Formatting.Indented)}");
logger.Queue($#"Encrypting config file passwords...");
bool updateConfig = false;
foreach (var kafkaSource in config.dataTargets)
{
if (!kafkaSource.password.IsEncrypted())
{
updateConfig = true;
logger.Queue($#"%tabEncrypting: {kafkaSource.name}");
kafkaSource.password = kafkaSource.password.Encrypt();
}
else
{
logger.Queue($#"%tabAlready encrypted: {kafkaSource.name}");
}
}
logger.Send(Environment.NewLine);
if (updateConfig)
{
File.WriteAllText(
configFilePath,
Newtonsoft.Json.JsonConvert.SerializeObject(config));
}
var _source = config.dataSources.FirstOrDefault(x=>x.name==currentProfile.dataSource);
var _target = config.dataTargets.FirstOrDefault(x => x.name == currentProfile.dataTarget);
source = new Connectors.Sql(logger,
_source?.name,
_source?.connectionString,
_source.pollingInterval,
_source.maxRowsPerSelect,
_source.maxRowsPerUpdate);
target = new Connectors.KafkaProducer(logger)
{
bootstrapServers = _target?.bootstrapServers,
name = _target?.name,
password = _target?.password.Decrypt(),
sslCaLocation = Path.Combine(assemblyLocation,_target?.sslCaLocation),
topic = _target?.topic,
username = _target?.username
};
Start();
}
public void Start()
{
Timer timer = new Timer();
try
{
logger.Log($#"SQL polling interval: {source.pollingInterval} seconds");
timer.Interval = source.pollingInterval * 1000;
timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
timer.Start();
if (currentProfile.executeOnStartup)
Run();
}
catch (Exception ex)
{
var sb = new StringBuilder();
sb.AppendLine($#"Critical error encountered loading external service: {config.serviceName}.");
if (!timer.Enabled)
sb.AppendLine($#"service unloaded - Schedule not started!");
else
sb.AppendLine($#"service appears to be loaded and running on schedule.");
logger.Log(sb.ToString(), LOG.LEVEL.CRITICAL, ex);
}
}
public void OnTimer(object sender, ElapsedEventArgs e)
{
try
{
Run();
}
catch (Exception ex)
{
logger.Log($#"Critical error during scheduled run on service: {config.serviceName}.", LOG.LEVEL.CRITICAL, ex);
}
}
public async void Run()
{
//Get new alarm events from SQL source
logger.Queue("Looking for new alarms...");
var rows = await GetNewEvents();`
The exception occurred during the GetNewEvents method, which attempted to open a SqlConnection to a SQL server that was unavailable due to network issues, that method intentionally throws an exception, which should throw up to OnTimer, where it gets caught, logged, and the timer keeps running. During development/testing, I used invalid credentials, bad connection string, etc and simulated this type of error and it worked as expected, logged the error, kept running. For some reason recently, that error is not caught in OnTimer, it propagates up, where it should be caught by Start (but isn't), after that it should be caught by the parent service manager which is entirely wrapped in a try/catch with no throw's, and above that (because their could be multiple microservices managed by that service) the entry point to the service manager is wrapped in try/catch with no throws, all for isolation from microservice errors. For some reason though, now, the error from a VERY downstream application is propagating all the way up.
Typically, this code runs 24/7 no issues, the microservice it is loading from the config file launches and runs fine. The entry into that specific microservice starts with a try {...} catch (Exception ex) {...} block.
The concept is to have a microservice manager than can launch a number of microservices without having to install all of them as windows services, and have some level of configuration driven by a config file that dictates how the main service runs.
The microservice represented here opens a SQL connection, reads data, performs business logic, publishes results to Kafka, it does this on a polling interval dictated by the config file contained in the microservice. As stated above, its ran for months without issue.
Recently, I noticed the main microservice manager service was not running on the windows server, I investigated the Server Application Logs and found a "Runtime Error" that essentially stated the microservice, while attempting to connect to sql, failed (network issue) and caused the entire microservice manager to exit. To my understanding, they way I'm launching the microservice should isolate it from the main service manager app. Additionally, the main service manager app is wrapped in a very generic try catch block. The entry point to the micro service itself is wrapped in a try catch, and almost every component in the microservice is wrapped in try / catch per business need. The scenario that faulted (cant connect to sql) intentionally throws an error for logging purposes, but should be caught by the immediate parent try/catch, which does not propagate or re-throw, only logs the error to a txt file and the windows server app log.
How is it that this exception is bubbling up through isolation points and causing the main service to fault and exit? I tested this extensively during development and prior to release, this exact scenario being unable to connect to sql, and it generated the correct log entry, and tried again on the next polling cycle as expected.
I haven't tried any other approaches as yet, as I feel they would be band-aid fixes as best as I dont understand why the original design is suddenly failing. The server hasn't changed, no patching/security updates/etc.
From the server Application Log:
Application: DataTransportService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
at Connectors.SqlHelper.DbHelper+d__13`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__14.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(System.Threading.Tasks.Task)
at IntelligentAlarms.IntelligentAlarm+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_1(System.Object)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

windows service will not stop when I select stop from the services windows C#

I have written a windows service but when I try to stop the service it says that the service cannot be stopped at this time. Here's my whole class:
public partial class RenewalsService : ServiceBase
{
private readonly ManualResetEvent _shutdownEvent = new ManualResetEvent(false);
private Thread _thread;
public RenewalsService()
{
InitializeComponent();
this.CanStop = true;
}
protected override void OnStart(string[] args)
{
_thread = new Thread(WorkerThread)
{
Name = "Renewals Service Thread",
IsBackground = true
};
_thread.Start();
}
protected override void OnStop()
{
try
{
if (!_shutdownEvent.SafeWaitHandle.IsClosed)
{
_shutdownEvent.Set();
}
if (_thread.IsAlive)
{
if (!_thread.Join(3000))
{
// give the thread 3 seconds to stop
_thread.Abort();
}
}
}
catch (Exception ex)
{
// _thread.Join may raise an error at this point. If it does we dont care. We dont care about any other exceptions
// since we are already in the process of closing the service.
}
finally
{
IError logger = new Logger();
Exception ex = new Exception("The Renewals service has been stopped.");
logger.Log(this, SeverityEnum.Warning, ex);
Environment.ExitCode = 0;
Environment.Exit(Environment.ExitCode);
}
}
private void WorkerThread()
{
try
{
while (!_shutdownEvent.WaitOne(1))
{
string timeToRun = ConfigurationManager.AppSettings["RunTime"];
string[] timeStrings = timeToRun.Split(':');
TimeSpan runtime = new TimeSpan(0, Int32.Parse(timeStrings[0]), Int32.Parse(timeStrings[1]), Int32.Parse(timeStrings[2]));
if (DateTime.Today.TimeOfDay.Hours == runtime.Hours &&
DateTime.Today.TimeOfDay.Minutes == runtime.Minutes)
{
Renewals renewals = new Renewals();
renewals.GenerateRenewal();
}
}
}
catch (Exception ex)
{
IError logger = new Logger();
logger.Log(this, SeverityEnum.Warning, ex);
this.OnStop();
}
}
}
What's missing to ensure the user can stop the service.

Your code looks ok to me, so here's a couple of things to check.
First, does the GenerateRenewal() method take a long time to complete? If so, you might need to periodically check _shutdownEvent inside that method for a timely shutdown. Of course, you've marked the thread as a background thread so it should shut down when you tell the service to stop anyway. I haven't seen background threads hold up process termination, but I guess there's always that chance.
Second, the more likely culprit to me is that the service has already shut down due to an exception. The Services console doesn't automatically refresh when a service shuts down, so it's possible you see the Stop link available to you when it shouldn't be. If you hit F5, the console will refresh, and if your service has stopped, the Start link should be the only one available. Check your log files to see if your exception handlers have been triggered.
UPDATE
So it looks like your WorkerThread() method is throwing an exception, which causes the service to stop. This explains why the Stop link is giving you the error message when you click it.
Providing you have sufficient permissions on your box, use this link to debug your service to find out why the exception is occurring.
HTH

The base ServiceBase class calls your overridden virtual method OnStop() when the Windows Service Control Manager ("the SCM") has sent the service a "Stop" command. In the method's implementation you are supposed to do whatever is necessary to get your service to a stopped state, then return from the method back to the ServiceBase class, which handles the interaction with the SCM, in this case to tell the SCM that your service is now stopped. The SCM will decide when your service process should be terminated, and the ServiceBase class handles that without you needing to do anything explicit.
For a well-behaved service, you should either just return at the end of your OnStop method, or throw an exception. The ServiceBase class will handle things appropriately, including logging your exception, if you have thrown one, as an error in the Windows Event Log. If your method may take a while to get your service stopped, you should call base.RequestAdditionalTime() at the appropriate points, so the base class can tell the SCM that you haven't just hung, your service is in the process of stopping.
I think your main problem lies in these lines:
Environment.ExitCode = 0;
Environment.Exit(Environment.ExitCode);
You never return to the base class at all... so the ServiceBase class never has a chance to respond gracefully to the SCM... you are just unilaterally terminating the process hosting your service. This is not what a well-behaved Windows service does.
The ServiceBase class is designed to be able to support multiple services hosted in a single service process. Individual services should not concern themselves with the lifetime of the host service process, only with the logical state of their own service.

WCF Windows Service Error: The socket connection has been disposed

I have 2 WCF services that are both installed locally on the same Windows Service. The 2 WCF services make calls to the same database and use Net TCP Binding.
Side Note: I made 2 services instead of 1 because the service had too many methods and as a result, the XML file was too large and I couldn't add the service reference to the consumer app. Also, it's more readable this way.
On the client app, I've added a service reference to both WCF services. To use the services, I extended the CommunicationObject so it has a "Using()" method: see the code below.
Communication Object Extension for .Using() Method
public static class CommunicationObjectExtensions
{
public static void Using<T>(this T client, Action<T> work)
where T : ICommunicationObject
{
try
{
// make sure client is open
//client.Open(); // didn't work
work(client);
}
catch (CommunicationException e)
{
client.Abort();
throw;
}
catch (TimeoutException e)
{
client.Abort();
throw;
}
catch (Exception e)
{
client.Abort();
throw;
}
finally
{
client.Close();
}
}
}
Service Calls
new ServiceClientA().Using(client =>
{
var someObjectA = client.GetSomeObjectA();
});
new ServiceClientB().Using(client =>
{
var someObjectB = client.GetSomeObjectB();
});
I get the "The socket connection has been disposed" error when ServiceClientB's method actually has data to return (versus an empty list) and returns (I'm able to debug and go through the entire service call).
I don't run into this problem when using the other WCF services that point to different databases.
Anyone have any ideas what's going on?
Update
Ok turns out it only happens with the method I'm calling (client.GetSomeObjectB()). The object it's returning is causing errors. Apparently this is because the object is too big, but it shouldn't be a problem (since I'm only returning 1 small object).

How to handle WCF exceptions (consolidated list with code)

I'm attempting to extend this answer on SO to make a WCF client retry on transient network failures and handle other situations that require a retry such as authentication expiration.
Question:
What are the WCF exceptions that need to be handled, and what is the correct way to handle them?
Here are a few sample techniques that I'm hoping to see instead of or in addition to proxy.abort():
Delay X seconds prior to retry
Close and recreate a New() WCF client. Dispose the old one.
Don't retry and rethrow this error
Retry N times, then throw
Since it's unlikely one person knows all the exceptions or ways to resolve them, do share what you know. I'll aggregate the answers and approaches in the code sample below.
// USAGE SAMPLE
//int newOrderId = 0; // need a value for definite assignment
//Service<IOrderService>.Use(orderService=>
//{
// newOrderId = orderService.PlaceOrder(request);
//}
/// <summary>
/// A safe WCF Proxy suitable when sessionmode=false
/// </summary>
/// <param name="codeBlock"></param>
public static void Use(UseServiceDelegateVoid<T> codeBlock)
{
IClientChannel proxy = (IClientChannel)_channelFactory.CreateChannel();
bool success = false;
try
{
codeBlock((T)proxy);
proxy.Close();
success = true;
}
catch (CommunicationObjectAbortedException e)
{
// Object should be discarded if this is reached.
// Debugging discovered the following exception here:
// "Connection can not be established because it has been aborted"
throw e;
}
catch (CommunicationObjectFaultedException e)
{
throw e;
}
catch (MessageSecurityException e)
{
throw e;
}
catch (ChannelTerminatedException)
{
proxy.Abort(); // Possibly retry?
}
catch (ServerTooBusyException)
{
proxy.Abort(); // Possibly retry?
}
catch (EndpointNotFoundException)
{
proxy.Abort(); // Possibly retry?
}
catch (FaultException)
{
proxy.Abort();
}
catch (CommunicationException)
{
proxy.Abort();
}
catch (TimeoutException)
{
// Sample error found during debug:
// The message could not be transferred within the allotted timeout of
// 00:01:00. There was no space available in the reliable channel's
// transfer window. The time allotted to this operation may have been a
// portion of a longer timeout.
proxy.Abort();
}
catch (ObjectDisposedException )
{
//todo: handle this duplex callback exception. Occurs when client disappears.
// Source: https://stackoverflow.com/questions/1427926/detecting-client-death-in-wcf-duplex-contracts/1428238#1428238
}
finally
{
if (!success)
{
proxy.Abort();
}
}
}

EDIT: There seems to be some inefficiencies with closing and reopening the client multiple times. I'm exploring solutions here and will update & expand this code if one is found. (or if David Khaykin posts an answer I'll mark it as accepted)
After tinkering around with this for a few years, the code below is my preferred strategy (after seeing this blog posting from the wayback machine) for dealing with WCF retries and handling exceptions.
I investigated every exception, what I would want to do with that exception, and noticed a common trait; every exception that needed a "retry" inherited from a common base class. I also noticed that every permFail exception that put the client into an invalid state also came from a shared base class.
The following example traps every WCF exception a client could through, and is extensible for your own custom channel errors.
Sample WCF Client Usage
Once you generate your client side proxy, this is all you need to implement it.
Service<IOrderService>.Use(orderService=>
{
orderService.PlaceOrder(request);
}
ServiceDelegate.cs
Add this file to your solution. No changes are needed to this file, unless you want to alter the number of retries or what exceptions you want to handle.
public delegate void UseServiceDelegate<T>(T proxy);
public static class Service<T>
{
public static ChannelFactory<T> _channelFactory = new ChannelFactory<T>("");
public static void Use(UseServiceDelegate<T> codeBlock)
{
IClientChannel proxy = null;
bool success = false;
Exception mostRecentEx = null;
int millsecondsToSleep = 1000;
for(int i=0; i<5; i++) // Attempt a maximum of 5 times
{
// Proxy cann't be reused
proxy = (IClientChannel)_channelFactory.CreateChannel();
try
{
codeBlock((T)proxy);
proxy.Close();
success = true;
break;
}
catch (FaultException customFaultEx)
{
mostRecentEx = customFaultEx;
proxy.Abort();
// Custom resolution for this app-level exception
Thread.Sleep(millsecondsToSleep * (i + 1));
}
// The following is typically thrown on the client when a channel is terminated due to the server closing the connection.
catch (ChannelTerminatedException cte)
{
mostRecentEx = cte;
proxy.Abort();
// delay (backoff) and retry
Thread.Sleep(millsecondsToSleep * (i + 1));
}
// The following is thrown when a remote endpoint could not be found or reached. The endpoint may not be found or
// reachable because the remote endpoint is down, the remote endpoint is unreachable, or because the remote network is unreachable.
catch (EndpointNotFoundException enfe)
{
mostRecentEx = enfe;
proxy.Abort();
// delay (backoff) and retry
Thread.Sleep(millsecondsToSleep * (i + 1));
}
// The following exception that is thrown when a server is too busy to accept a message.
catch (ServerTooBusyException stbe)
{
mostRecentEx = stbe;
proxy.Abort();
// delay (backoff) and retry
Thread.Sleep(millsecondsToSleep * (i + 1));
}
catch (TimeoutException timeoutEx)
{
mostRecentEx = timeoutEx;
proxy.Abort();
// delay (backoff) and retry
Thread.Sleep(millsecondsToSleep * (i + 1));
}
catch (CommunicationException comException)
{
mostRecentEx = comException;
proxy.Abort();
// delay (backoff) and retry
Thread.Sleep(millsecondsToSleep * (i + 1));
}
catch(Exception e)
{
// rethrow any other exception not defined here
// You may want to define a custom Exception class to pass information such as failure count, and failure type
proxy.Abort();
throw e;
}
}
if (success == false && mostRecentEx != null)
{
proxy.Abort();
throw new Exception("WCF call failed after 5 retries.", mostRecentEx );
}
}
}

I started a project on Codeplex that has the following features
Allows efficient reuse of the client proxy
Cleans up all resources, including EventHandlers
Operates on Duplex channels
Operates on Per-call services
Supports config constructor, or by factory
http://smartwcfclient.codeplex.com/
It is a work in progress, and is very heavily commented. I'll appreciate any feedback regarding improving it.
Sample usage when in instance mode:
var reusableSW = new LC.Utils.WCF.ServiceWrapper<IProcessDataDuplex>(channelFactory);
reusableSW.Reuse(client =>
{
client.CheckIn(count.ToString());
});
reusableSW.Dispose();

we have a WCF client that deal with almost any type of failure at the server. The Catch list is very long but does not have to be. If you look closely, you will see that many exceptions are child definitions of the Exception Class (and a few other classes).
Thus you can simplify things a lot if you want to. That said, here are some typical errors that we catch:
Server timeout
Server too busy
Server unavailable.

Below links may help to handle WCF Exceptions:
http://www.codeproject.com/KB/WCF/WCFErrorHandling.aspx
http://msdn.microsoft.com/en-us/library/cc949036.aspx

MessageQueue.BeginReceive() null ref error - c#

Have a windows service that listens to a msmq. In the OnStart method is have this
protected override void OnStart(string[] args)
{
try
{
_queue = new MessageQueue(_qPath);//this part works as i had logging before and afer this call
//Add MSMQ Event
_queue.ReceiveCompleted += new ReceiveCompletedEventHandler(queue_ReceiveCompleted);//this part works as i had logging before and afer this call
_queue.BeginReceive();//This is where it is failing - get a null reference exception
}
catch(Exception ex)
{
EventLogger.LogEvent(EventSource, EventLogType, "OnStart" + _lineFeed +
ex.InnerException.ToString() + _lineFeed + ex.Message.ToString());
}
}
where
private MessageQueue _queue = null;
This works on my machine but when deployed to a windows 2003 server and running as Network service account, it fails
Exception recvd:
Service cannot be started. System.NullReferenceException: Object reference not set to an instance of an object.
at MYService.Service.OnStart(String[] args)
at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)
Solved:
Turned out that the Q that i set up, I had to explicitly add Network Service account to it under security tab

You're seeing that particular exception because you're calling ex.InnerException.ToString(). The InnerException property is not always populated (in fact, it frequently isn't, nor should it be).
Your root problem is likely that the Network Service account doesn't have permissions to access the queue (in this case, read from it).
Here's some code that will help you get the actual error in your event log:
catch(Exception ex)
{
Exception e = ex;
StringBuilder message = new StringBuilder();
while(e != null)
{
if(message.Length > 0) message.AppendLine("\nInnerException:");
message.AppendLine(e.ToString());
e = e.InnerException;
}
EventLogger.LogEvent(EventSource, EventLogType, "OnStart" + _lineFeed +
message.ToString());
}