We Keep Coding

WCF Service hosted in Windows Service hangs on stop

I have a WCF Service hosted in a Windows service as described here.
I have scheduled nightly restart of the service, but sometimes the restart fails and the service remains/hangs in Stopping state and the EXE process has to be killed manually. It looks likely that it hangs on line _ESSServiceHost.Close();, because nothing after that line is logged it the log file. It is possible but not very likely that the service gets the stop request when it is busy.
Moreover the underlying process cannot be killed because it is dependent on services.exe, so only server restart works.
What could be wrong with this approach?
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
_ESSServiceHost.Close();
_ESSServiceHost = null;
//Never reaches the following line
Tools.LogInfo("Services stopped.");
}
}
catch (Exception ex)
{
Tools.LogError(ex.Message);
}
This is how I stop the service:
private bool StopService(ServiceController scESiftServer)
{
int i = 0;
if (scESiftServer.Status == ServiceControllerStatus.Running)
{
try
{
scESiftServer.Stop();
}
catch (Exception ex)
{
Tools.LogEvent("Exception ...");
return false;
}
while (scESiftServer.Status != ServiceControllerStatus.Stopped && i < 120)
{
Thread.Sleep(1000);
scESiftServer.Refresh();
i++;
}
}
if (scESiftServer.Status != ServiceControllerStatus.Stopped)
{
//This line gets executed
Tools.LogEvent("Failed within 120 sec...");
return false;
}
else
{
Tools.LogEvent("OK ...");
}
return true;
}
Could something like this help?
var task = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(299)));
if (!task.Wait(TimeSpan.FromSeconds(300)))
{
_ESSServiceHost.Abort();
}
But _ESSServiceHost.Abort() should be called internally by the Close method if needed.
Target framework is 4.5, installed is .NET 4.7.2.
Found out that probably the service hangs after series of malformed requests. Expected record type 'Version', found '71'. etc.

I have found in the svclog file that my service hangs after series of malformed request that happen on Saturday and Sunday at approx. 5:15 AM. The error messages were Expected record type 'Version', found '71'., Error while reading message framing format at position 0 of stream (state: ReadingVersionRecord). But I could not find the cause of theese malformed request series, so I tried to fix the service to withstand the "attack".
I have modified the OnStop method as follows:
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
Tools.LogInfo("Stopping ESService.");
var abortTask = Task.Run(() => _ESSServiceHost.Abort());
var closeTask = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(300)));
try
{
if (_ESSServiceHost.State == CommunicationState.Faulted)
{
Tools.LogInfo("ESSServiceHost.State == CommunicationState.Faulted");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
else
{
if (!closeTask.Wait(TimeSpan.FromSeconds(301)))
{
Tools.LogInfo("Failed to Close - trying Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
}
}
catch (Exception ex)
{
Tools.LogException(ex, "ESSServiceHost.Close");
try
{
Tools.LogInfo("Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
catch (Exception ex2)
{
Tools.LogException(ex2, "ESSServiceHost.Abort");
}
}
_ESSServiceHost = null;
Tools.LogInfo("ESService stopped.");
}
}
catch (Exception ex)
{
Tools.LogException(ex,"OnStop");
}
}
Today on Monday I have checked the svclog and the "attacks" with malformed request remained there but my service lived happily through it. So it seemed to be fixed. Moreover only:
Stopping ESService.
ESService stopped.
events were logged in my log file. No aborts etc. So I guess that putting the Close call on a separate thread fixed the problem but absolutely do not know why.

Windows Service is Running but not executing code

We have windows service which is running fine untill any exceptions occured in the process.
It contains two Threads (GenerateInvoice and GenerateReport).
These threads are getting blocked and results in DeadLock like situation mostly when there is high CPU usage on our DataBase server.
We have done some changes in code to handle such situations like added while condition below code but still it is not working.
Below is the OnStart() method of service:
protected override void OnStart(string[] args)
{
try
{
log.Debug("Starting Invoice Generation Service");
_thread = new Thread(new ThreadStart((new GenerateInvoice()).Process));
_thread.IsBackground = true;
_thread.Start();
_reportThread = new Thread(new ThreadStart((new GenerateReport()).Process));
_reportThread.IsBackground = true;
_reportThread.Start();
}
catch (Exception ex)
{
log.Error("Error in Invoice Generation Service:", ex);
}
}
Here is the processing code of first thread: GenerateInvoice
public void Process()
{
while (isProcessActive)
{
try
{
DBBilling obj = new DBBilling();
DataTable dtInvoiceID = obj.readData(#"SELECT * FROM (SELECT ird.BillByType, ird.InvoiceID, ir.BeginDate, ir.EndDate, ir.SendToQB, ir.SendEmail,
i.ARAccountID, i.ARAccountHotelID, i.invoiceNumber,i.[STATUS],UPDATETIME,row_number() over (PARTITION BY ird.INVOICEID ORDER BY UPDATETIME DESC) AS row_number
FROM Invoices i JOIN InvoicesRunRequestDetails ird ON ird.InvoiceID=i.InvoiceID
JOIN InvoicesRunRequest ir ON ird.RequestID = ir.RequestID
Where i.[STATUS] = 'PENDING') AS rows
WHERE ROW_NUMBER=1 ORDER BY UPDATETIME");
processCounter = 0;
#region process
if (dtInvoiceID != null && dtInvoiceID.Rows.Count > 0)
{
//some code here..
}
#endregion
}
catch (Exception ex) //Mantis 1486 : WEBPMS1 Disk Space : 10 Aug 2016
{
log.ErrorFormat("Generate Invoice -> Process -> InnLink Billing Execute Query Exception. Error={0}", ex);
if(DBBilling.dbConnTimeoutErrorMessage.Any(ex.Message.Contains))
{
processCounter++;
if (processCounter >= 1) //Need to change to 25 after Problem Solve
{
isProcessActive = false;
log.ErrorFormat("Generate Invoice -> Process -> RunInvoice Service exiting loop"); //From here control is not going back
}
else
System.Threading.Thread.Sleep(5000); //Sleep for 5 Sec
}
}
}
}
Processing of Second Thread i.e. GenerateReport code:
public void Process()
{
AppSettingsReader ar = new AppSettingsReader();
string constr = (string)ar.GetValue("BillingDB", typeof(string));
SqlConnection con = new SqlConnection(constr);
while (isProcessActive)
{
try
{
DBBilling obj = new DBBilling();
DataTable dtReportRunID = obj.readData(#"SELECT ReportRunID,MonYear, BeginDate, EndDate FROM ReportRunRequest
Where [STATUS] = 'PENDING' ORDER BY ReportRunID");
processCounter = 0;
if (dtReportRunID != null && dtReportRunID.Rows.Count > 0)
{
//some code here..
}
}
catch (Exception ex) //Mantis 1486 : WEBPMS1 Disk Space : 10 Aug 2016
{
log.ErrorFormat("Generate Report -> Process -> InnLink Billing Execute Query Exception. Error={0}", ex);
if (DBBilling.dbConnTimeoutErrorMessage.Any(ex.Message.Contains))
{
processCounter++;
if (processCounter >= 1) //Need to change to 25 after Problem Solve
{
isProcessActive = false;
log.ErrorFormat("Generate Report -> Process -> RunInvoice Service Exiting loop"); //From here control is not going back
}
else
System.Threading.Thread.Sleep(5000); //Sleep for 5 Sec
}
}
}
}
What possible solution to avoid such conditions?

The way to avoid it is to either lock every access to a global variable, or not to use global variables.
here is one obvious example
DBBilling.dbConnTimeoutErrorMessage.Any(ex.Message.Contains)
dbConnTimeoutErrorMessage is a static field that is being used from two different threads and I assume is not thread safe, surround access to it with a
lock(locObj)
{
// access to dbConnTimeoutErrorMessage
}
I am gonna go ahead and guess that log is also a global variable. Perhaps maybe even isProcessActive or processCounter.
I am guessing there is more in those comments - make sure your code is threadsafe before using it with two different threads.
I doubt locking access to what I said will fix your problem, but I guess your lack of threadsafe programming in these is a symptom to not using lock when it is needed. The secret is to lock every access to a global context, and just that.

What i suggest is to use Timer instead of infinite loop and as mentioned earlier in other answere you need some kind of synchronization. First of all, you need to implement your variables which used in different threads as follows (i don't know exactly definitions of your variables, but main idea is to use volatile keyword in your case):
public static volatile bool isProcessActive;
public static volatile int proccessCounter;
volatile keyword switches off the compiler optimizations for using variable in one thread. It means that your variables now are thread safe.
Next you need to use neither System.Threading.Timer or System.Timers.Timer. I will use in my example second one.
public sealed class GenerateInvoice :
{
protected const int timerInterval = 1000; // define here interval between ticks
protected Timer timer = new Timer(timerInterval); // creating timer
public GenerateInvoice()
{
timer.Elapsed += Timer_Elapsed;
}
public void Start()
{
timer.Start();
}
public void Stop()
{
timer.Stop();
}
public void Timer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
DBBilling obj = new DBBilling();
DataTable dtInvoiceID = obj.readData(#"SELECT * FROM (SELECT ird.BillByType, ird.InvoiceID, ir.BeginDate, ir.EndDate, ir.SendToQB, ir.SendEmail,
i.ARAccountID, i.ARAccountHotelID, i.invoiceNumber,i.[STATUS],UPDATETIME,row_number() over (PARTITION BY ird.INVOICEID ORDER BY UPDATETIME DESC) AS row_number
FROM Invoices i JOIN InvoicesRunRequestDetails ird ON ird.InvoiceID=i.InvoiceID
JOIN InvoicesRunRequest ir ON ird.RequestID = ir.RequestID
Where i.[STATUS] = 'PENDING') AS rows
WHERE ROW_NUMBER=1 ORDER BY UPDATETIME");
processCounter = 0;
#region process
if (dtInvoiceID != null && dtInvoiceID.Rows.Count > 0)
{
//some code here..
}
#endregion
}
catch (Exception ex) //Mantis 1486 : WEBPMS1 Disk Space : 10 Aug 2016
{
log.ErrorFormat("Generate Invoice -> Process -> InnLink Billing Execute Query Exception. Error={0}", ex);
if(DBBilling.dbConnTimeoutErrorMessage.Any(ex.Message.Contains))
{
processCounter++;
if (processCounter >= 1) //Need to change to 25 after Problem Solve
{
isProcessActive = false;
// supposing that log is a reference type and one of the solutions can be using lock
// in that case only one thread at the moment will call log.ErrorFormat
// but better to make synchronization stuff unside logger
lock (log)
log.ErrorFormat("Generate Invoice -> Process -> RunInvoice Service exiting loop"); //From here control is not going back
}
else
// if you need here some kind of execution sleep
// here you can stop timer, change it interval and run again
// it's better than use Thread.Sleep
// System.Threading.Thread.Sleep(5000); //Sleep for 5 Sec
}
}
}
}
Use the same approach for the GenerateReport to make Timer-based.
And, finally, you need to change your OnStart and OnStop methods something like so:
protected GenerateInvoice generateInvoice;
protected GenerateReport generateReport;
protected override void OnStart(string[] args)
{
// all exception handling should be inside class
log.Debug("Starting Invoice Generation Service");
generateInvoice = new GenerateInvoice();
generateInvoice.Start();
generateReport = new GenerateReport();
generateReport.Start();
}
protected override void OnStop()
{
generateInvoice.Stop();
generateReport.Stop();
}

System.Threading.Task not starting in production environment

I have a c# windows service which is doing various tasks. Its working perfectly on my local system but as soon as I start it on my product server, its doesn't perform a particular task on it.
This is how my service is structured:
public static void Execute()
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
string cc = dr["ccode"].ToString();
Task objTask = new Task(delegate { RequestForEachCustomer(cc); });
objTask.Start();
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestForEachCustomer(object cc)
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
WriteLog("RequestForEachCustomer - Before Task");
Task objTask = new Task(delegate { RequestProcessing(dr); });
objTask.Start();
WriteLog("RequestForEachCustomer - After Task");
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestProcessing(object dr)
{
try
{
WriteLog("Inside RequestProcessing");
// .... some work ....
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
Now what happens on the production server is that it logs the last entry in RequestForEachCustomer which is "RequestForEachCustomer - After Task" but it doesn't log the entry from RequestProcessing which mean the task is not starting at all. There are no exceptions in either database or text file.
There are no events logged in window's event viewer either. Also the service keeps working (if I insert another record in database, its processed by the service immediately so the service isn't stuck either. It just doesn't seem to process RequestProcessing task.)
I am baffled by this and it would be great if someone could point out the mistake I am making. Oh, btw did I forgot to mention that this service was working perfectly few days ago on the server and it is still working fine on my local PC.
EDIT :
WriteLog :
public static void WriteErrorLog(string Message)
{
StreamWriter sw = null;
try
{
lock (locker)
{
sw = new StreamWriter(AppDomain.CurrentDomain.BaseDirectory + "\\Logs\\LogFile.txt", true);
sw.WriteLine(DateTime.Now.ToString() + ": " + Message);
sw.Flush();
sw.Close();
}
}
catch (Exception excep)
{
try
{
// .... Inserting ErrorLog in DB ....
}
catch
{
throw excep;
}
throw excep;
}
}
I have also logged an entry on OnStop() something like "Service Stopped" and its logs every time I stop my service so the problem couldn't exist in WriteLog function.

I suggest you refactor your code as in this MSDN example. What bother me in your code is, you never wait for tasks to finish anywhere.
The following example starts 10 tasks, each of which is passed an index as a state object. Tasks with an index from two to five throw exceptions. The call to the WaitAll method wraps all exceptions in an AggregateException object and propagates it to the calling thread.
Source : Task.WaitAll Method (Task[])
This line from example might be of some importance :
Task.WaitAll(tasks.ToArray());

TcpListener.AcceptSocket() fails in a Windows Service

I have an instance of the following code that executes correctly in Debug or as a standalone Windows application:
TcpListener tcpListener = new TcpListener(IPAddress.Any, 4554);
tcpListener.Start();
while (true)
{
try
{
using (Socket socket = tcpListener.AcceptSocket())
{
// Code here is reached in Debug or as a Console Application
// but not as a Windows Service
}
}
catch (SocketException se)
{
// This is never reached
}
catch (Exception ex)
{
// This is never reached
}
finally
{
// This is never reached in the Windows Service
}
}
However, when I install it as a Windows Service, it crashes on tcpListener.AcceptSocket(), and logs the following to the Event Viewer:
An unhandled exception ('System.Net.Sockets.SocketException') occurred in MyService.exe [768]. Just-In-Time debugging this exception failed with the following error: The operation attempted is not supported.
Even trying to catch the exception I am unable to log anything more. Stepping through code in Debug accomplishes nothing because the code successfully blocks the application and waits for a client connection.
Is there a way to implement this for a Windows Service?

usr's advice (and this answer) led me to a bug in the code. The ServiceBase class contained the following:
protected override void OnStart(string[] args)
{
_worker = new Thread(ExecuteService);
_worker.Start();
}
private void ExecuteService()
{
for (;;)
{
if (_stop.WaitOne(1000))
{
new TcpServer().StartTcpServer();
return;
}
}
}
The correct implementation was to remove the for loop, which was re-instantiating the listener. Here is the final code:
protected override void OnStart(string[] args)
{
_worker = new Thread(ExecuteService);
_worker.Start();
}
private static void ExecuteService()
{
new TcpServer().StartTcpServer();
}

Bing Maps GetRoute gives '0x8004231C' error

I'm trying to show a route from point-to-point on the bing-maps (testing on real device). I've entered 2 waypoints (GeoCoordinate) and I'm trying to get the route via the Windows PhoneToolKit using the await query.GetRouteAsync(). Unfortunately, I'm getting an unknown error:
The result of the async call:
'e.Result' threw an exception of type 'System.Reflection.TargetInvocationException'
The inner exception:
Exception from HRESULT: 0x8004231C
I've checked the MSDN website and noticed that this errorcode is not listed in the errorlist...
The related code is below. I've used the exact same code as in the sample set of the Windows Phone Toolkit, but removed the things which has nothing to do with getting the route:
private async void BtnShowRoute_Tap(object sender, System.Windows.Input.GestureEventArgs e)
{
try
{
RouteQuery query = new RouteQuery();
List<GeoCoordinate> wayPoints = new List<GeoCoordinate>();
wayPoints.Add(new GeoCoordinate(47.23449, -121.172447));
wayPoints.Add(new GeoCoordinate(47.062638, -120.691795));
query.Waypoints = wayPoints;
Route route = await query.GetRouteAsync();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
throw;
}
}
I have no idea what is going wrong here. Does anyone else experienced this issue? If so, did you resolve it? And how?
Note: I'm running Windows Phone 8.1. Dev Preview

This happens when the underlying service call times out before completing the query. Hopefully this will be fixed in next version , but for now you can use following code:
private async void BtnShowRoute_Tap(object sender, System.Windows.Input.GestureEventArgs e)
{
RouteQuery query = new RouteQuery();
List<GeoCoordinate> wayPoints = new List<GeoCoordinate>();
wayPoints.Add(new GeoCoordinate(47.23449, -121.172447));
wayPoints.Add(new GeoCoordinate(47.062638, -120.691795));
query.Waypoints = wayPoints;
query .QueryCompleted += geoQ_QueryCompleted;
query.GetRouteAsync();
}
private void geoQ_QueryCompleted(object sender, QueryCompletedEventArgs<Route> e)
{
try
{
Route myRoute = e.Result;
}
catch (TargetInvocationException)
{
Thread.Sleep(1000); // waiting for completing the query
geoQ_QueryCompleted(sender, e);
}
}