We Keep Coding

WCF Service hosted in Windows Service hangs on stop

I have a WCF Service hosted in a Windows service as described here.
I have scheduled nightly restart of the service, but sometimes the restart fails and the service remains/hangs in Stopping state and the EXE process has to be killed manually. It looks likely that it hangs on line _ESSServiceHost.Close();, because nothing after that line is logged it the log file. It is possible but not very likely that the service gets the stop request when it is busy.
Moreover the underlying process cannot be killed because it is dependent on services.exe, so only server restart works.
What could be wrong with this approach?
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
_ESSServiceHost.Close();
_ESSServiceHost = null;
//Never reaches the following line
Tools.LogInfo("Services stopped.");
}
}
catch (Exception ex)
{
Tools.LogError(ex.Message);
}
This is how I stop the service:
private bool StopService(ServiceController scESiftServer)
{
int i = 0;
if (scESiftServer.Status == ServiceControllerStatus.Running)
{
try
{
scESiftServer.Stop();
}
catch (Exception ex)
{
Tools.LogEvent("Exception ...");
return false;
}
while (scESiftServer.Status != ServiceControllerStatus.Stopped && i < 120)
{
Thread.Sleep(1000);
scESiftServer.Refresh();
i++;
}
}
if (scESiftServer.Status != ServiceControllerStatus.Stopped)
{
//This line gets executed
Tools.LogEvent("Failed within 120 sec...");
return false;
}
else
{
Tools.LogEvent("OK ...");
}
return true;
}
Could something like this help?
var task = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(299)));
if (!task.Wait(TimeSpan.FromSeconds(300)))
{
_ESSServiceHost.Abort();
}
But _ESSServiceHost.Abort() should be called internally by the Close method if needed.
Target framework is 4.5, installed is .NET 4.7.2.
Found out that probably the service hangs after series of malformed requests. Expected record type 'Version', found '71'. etc.

I have found in the svclog file that my service hangs after series of malformed request that happen on Saturday and Sunday at approx. 5:15 AM. The error messages were Expected record type 'Version', found '71'., Error while reading message framing format at position 0 of stream (state: ReadingVersionRecord). But I could not find the cause of theese malformed request series, so I tried to fix the service to withstand the "attack".
I have modified the OnStop method as follows:
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
Tools.LogInfo("Stopping ESService.");
var abortTask = Task.Run(() => _ESSServiceHost.Abort());
var closeTask = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(300)));
try
{
if (_ESSServiceHost.State == CommunicationState.Faulted)
{
Tools.LogInfo("ESSServiceHost.State == CommunicationState.Faulted");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
else
{
if (!closeTask.Wait(TimeSpan.FromSeconds(301)))
{
Tools.LogInfo("Failed to Close - trying Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
}
}
catch (Exception ex)
{
Tools.LogException(ex, "ESSServiceHost.Close");
try
{
Tools.LogInfo("Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
catch (Exception ex2)
{
Tools.LogException(ex2, "ESSServiceHost.Abort");
}
}
_ESSServiceHost = null;
Tools.LogInfo("ESService stopped.");
}
}
catch (Exception ex)
{
Tools.LogException(ex,"OnStop");
}
}
Today on Monday I have checked the svclog and the "attacks" with malformed request remained there but my service lived happily through it. So it seemed to be fixed. Moreover only:
Stopping ESService.
ESService stopped.
events were logged in my log file. No aborts etc. So I guess that putting the Close call on a separate thread fixed the problem but absolutely do not know why.

System.Threading.Task not starting in production environment

I have a c# windows service which is doing various tasks. Its working perfectly on my local system but as soon as I start it on my product server, its doesn't perform a particular task on it.
This is how my service is structured:
public static void Execute()
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
string cc = dr["ccode"].ToString();
Task objTask = new Task(delegate { RequestForEachCustomer(cc); });
objTask.Start();
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestForEachCustomer(object cc)
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
WriteLog("RequestForEachCustomer - Before Task");
Task objTask = new Task(delegate { RequestProcessing(dr); });
objTask.Start();
WriteLog("RequestForEachCustomer - After Task");
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestProcessing(object dr)
{
try
{
WriteLog("Inside RequestProcessing");
// .... some work ....
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
Now what happens on the production server is that it logs the last entry in RequestForEachCustomer which is "RequestForEachCustomer - After Task" but it doesn't log the entry from RequestProcessing which mean the task is not starting at all. There are no exceptions in either database or text file.
There are no events logged in window's event viewer either. Also the service keeps working (if I insert another record in database, its processed by the service immediately so the service isn't stuck either. It just doesn't seem to process RequestProcessing task.)
I am baffled by this and it would be great if someone could point out the mistake I am making. Oh, btw did I forgot to mention that this service was working perfectly few days ago on the server and it is still working fine on my local PC.
EDIT :
WriteLog :
public static void WriteErrorLog(string Message)
{
StreamWriter sw = null;
try
{
lock (locker)
{
sw = new StreamWriter(AppDomain.CurrentDomain.BaseDirectory + "\\Logs\\LogFile.txt", true);
sw.WriteLine(DateTime.Now.ToString() + ": " + Message);
sw.Flush();
sw.Close();
}
}
catch (Exception excep)
{
try
{
// .... Inserting ErrorLog in DB ....
}
catch
{
throw excep;
}
throw excep;
}
}
I have also logged an entry on OnStop() something like "Service Stopped" and its logs every time I stop my service so the problem couldn't exist in WriteLog function.

I suggest you refactor your code as in this MSDN example. What bother me in your code is, you never wait for tasks to finish anywhere.
The following example starts 10 tasks, each of which is passed an index as a state object. Tasks with an index from two to five throw exceptions. The call to the WaitAll method wraps all exceptions in an AggregateException object and propagates it to the calling thread.
Source : Task.WaitAll Method (Task[])
This line from example might be of some importance :
Task.WaitAll(tasks.ToArray());

Unable to catch ServiceController.Start() exceptions in calling thread

How can i catch the exception that occurs when starting a windows service. I am unable to get the exception here in my below code even though i am throwing exception in the Onstart() method of the service.
public class InterOpIntegrationWinService : ServiceBase
{
protected override void OnStart(string[] args)
{
throw new InvalidOperationException(message);
}
}
Calling thread code
try
{
using (ServiceController controller = new ServiceController())
{
controller.ServiceName = objServiceConfig.ServiceName;
controller.Start();
System.Windows.Forms.Application.DoEvents();
//controller.WaitForStatus(ServiceControllerStatus.Running, new TimeSpan(0, 0, 15));
//controller.WaitForStatus(ServiceControllerStatus.Running);
//if (!string.IsNullOrEmpty(LogUtilities.ServiceOnStartException))
//{
// MessageBox.Show("Error with starting service : " + LogUtilities.ServiceOnStartException);
// LogUtilities.ServiceOnStartException = string.Empty;
//}
}
}
catch (System.InvalidOperationException InvOpExcep)
{
DisplayError(InvOpExcep.Message);
LogUtilities.DisplayMessage("Failed to start service. " + LogUtilities.ServiceOnStartException, InvOpExcep);
LogUtilities.ServiceOnStartException = string.Empty;
}
catch (Exception ex)
{
DisplayError(ex.Message);
LogUtilities.DisplayMessage("Failed to start service. " + LogUtilities.ServiceOnStartException, ex);
LogUtilities.ServiceOnStartException = string.Empty;
}
i check for application license in the onstart() method and throws a licensing error if it fails. i want this to shared to my calling thread so i could show the message in a DialogBox. Any ideas of how to do this if i cannot handle the exceptions in my calling process.

Separate your service into (at least) two components - a component that deals with IPC in some form (e.g. Remoting, WCF endpoint, REST service, etc) and (one or more) components that do its actual job.
If the licensing check fails, don't start the other components - but do still start the component that offers IPC. After starting your service (which should now always at least start), you forms-based application can connect to the service and (through whatever means you want) determine that the service is currently refusing to provide any functionality due to a failed licensing check.

Apply Stop on a service that is Stopped or has another state

I am using the following function to stop my windows service (with a timeout):
public int StopService()
{
ServiceController service = new ServiceController(serviceName);
try
{
TimeSpan timeout = TimeSpan.FromMilliseconds(timeoutMilliseconds);
service.Stop();
service.WaitForStatus(ServiceControllerStatus.Stopped, timeout);
return 1;
}
catch (Exception ex)
{
logger.Log(LoggingLevel.Error,
this.GetType().Name + ":" + MethodBase.GetCurrentMethod().Name,
string.Format("{0}", ex.Message));
return -1;
}
}
I check the status of the service. And if it is not ServiceControllerStatus.Stopped I call this function.
I have tested and the program works when the status of the server is Running or Stopped. But I would like to know what happens when the status is other states such as StopPending, StartPending, etc. If I tell my service to stop while the state is one of the above, will the Stop command still do its job?

It depends on the service and what it permits at the time of the call to Stop().
At the Win32 level, each service must regularly call SetServiceStatus() to inform the Service Control Manager (SCM) what commands it will accept. Your call to Stop() will only succeed when the service has explicitly said that it allows that operation. In my experience, services in a pending state do not usually allow the normal range of operations so a call to Stop() will likely fail.
Note that it is easy to check what is permitted in C#, as illustrated in this sample code under the Examples heading. You can prefix your call to Stop() with CanStop(), and maybe wait a while if the service isn't quite ready to be stopped yet. Contingencies abound.

You can try like below, below example for StopPending ,
while (service.Status == ServiceControllerStatus.StopPending) //If it is stop pending, wait for it
{
System.Threading.Thread.Sleep(30 * 1000); // thread sleep for 30 seconds
service.Refresh(); // refreshing status
if (service.Status == ServiceControllerStatus.Stopped)
{
Comments = "Service " + serviceName + " stopped successfully. ";
}
}
Let me know whether it works or not. Please also provide if you have(had) any better solution

Consumer service not dequeuing all messages from ActiveMQ queue

I have a windows service that is attempting to consume messages from some activemq queue's. However, it is only getting some of the messages and others are getting stuck in 'messages pending' in the queue. ActiveMQ tells me it has enqueued lets say 500 messages to the consumer but only 300 were dequeued. There is more than one listener being set up in the service. Here's the important part of the code:
private void setupListener(string queue, string brokerUri)
{
try
{
ISession session = connectionConsumers[brokerUri].CreateSession();
session.CreateConsumer(session.GetQueue(queue))
.Listener += new MessageListener(consumer_Listener);
}
catch (Exception ex)
{
Log.Error("An exception has occured setting up listener for " + queue + " on " + brokerUri + ": {0}, {1}", ex, ex.Message);
}
}
void consumer_Listener(IMessage message)
{
try
{
processLog((message as ITextMessage).Text);
message.Acknowledge();
}
catch (NMSException ex)
{
Log.Error("ActiveMQ Connection Failure: {0}, {1}", ex, ex.Message);
}
catch (Exception ex)
{
Log.Error("An exception has occured trying to process a message: {0}, {1}", ex, ex.Message);
}
}
Is there something wrong with the way I'm acknowledging messages that would cause certain ones to not be acknowledged? Is it a concurrency issue? I'm not sure if they are all still going through the processLog function (added to my database).
EDIT: I think it has more to do with acknowledgements not happening properly (for some reason). I am not getting exceptions thrown in my logs. However, activemq shows the following:
From what I've read, the dispatch queue is being filled with messages that were sent to the consumer but not acknowledged. Why could this be?

The problem had to do with our queues being virtual destinations.