Task.Run on web farm not running correctly - c#

I have a call to write to a log file whenever the API is called. This works flawlessly on single machine but as soon as it is moved to a web farm, then nothing gets written. No errors are raised either.
Here is how things are arranged
Call to API
[HttpGet]
public model.Returns Get([FromUri] model.Requests requests)
With this function, Get, is a call to write to the log file like so
//record request
Task.Run(() => writers.WriteApiLogAsync(requests));
Within the WriteApiLogAsync is this
logs.Add(new model.Log()
{
Type = requests.t,
When = DateTime.Now,
PhoneId = requests.p,
Location = requests.l,
Phone = requests.ph,
PhoneType = requests.pt
});
//file is locked, attempt write on next round but stack log entries
if (logLocker.IsWriterLockHeld) return;
try
{
var result = await writeLogs(logs);
if(result) logLocker.ReleaseWriterLock();
}
catch (OutOfMemoryException ex)
{
WriteErrorAsnc(ex.Message, ex.ToString(), "WriteApiLogAsync_OutOfMemoryException");
logs.Clear();
logLocker.ReleaseWriterLock();
}
catch (IOException ex)
{
WriteErrorAsnc(ex.Message, ex.ToString(), "WriteApiLogAsync_IOException");
logs.Clear();
logLocker.ReleaseWriterLock();
}
catch (Exception ex)
{
WriteErrorAsnc(ex.Message, ex.ToString(), "WriteApiLogAsync_Exception");
logs.Clear();
logLocker.ReleaseWriterLock();
}
finally
{
if (logs.Count > 1000)
{
logs.Clear();
logLocker.ReleaseWriterLock();
}
}
logs is set as
private static List<model.Log> logs = new List<model.Log>();
I didn't specify a machine key as I understand it is primarily for encryption/decryption which there isn't any.
Anyone have an idea of the cause or can point to somewhere with same issue?

Related

WCF Service hosted in Windows Service hangs on stop

I have a WCF Service hosted in a Windows service as described here.
I have scheduled nightly restart of the service, but sometimes the restart fails and the service remains/hangs in Stopping state and the EXE process has to be killed manually. It looks likely that it hangs on line _ESSServiceHost.Close();, because nothing after that line is logged it the log file. It is possible but not very likely that the service gets the stop request when it is busy.
Moreover the underlying process cannot be killed because it is dependent on services.exe, so only server restart works.
What could be wrong with this approach?
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
_ESSServiceHost.Close();
_ESSServiceHost = null;
//Never reaches the following line
Tools.LogInfo("Services stopped.");
}
}
catch (Exception ex)
{
Tools.LogError(ex.Message);
}
This is how I stop the service:
private bool StopService(ServiceController scESiftServer)
{
int i = 0;
if (scESiftServer.Status == ServiceControllerStatus.Running)
{
try
{
scESiftServer.Stop();
}
catch (Exception ex)
{
Tools.LogEvent("Exception ...");
return false;
}
while (scESiftServer.Status != ServiceControllerStatus.Stopped && i < 120)
{
Thread.Sleep(1000);
scESiftServer.Refresh();
i++;
}
}
if (scESiftServer.Status != ServiceControllerStatus.Stopped)
{
//This line gets executed
Tools.LogEvent("Failed within 120 sec...");
return false;
}
else
{
Tools.LogEvent("OK ...");
}
return true;
}
Could something like this help?
var task = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(299)));
if (!task.Wait(TimeSpan.FromSeconds(300)))
{
_ESSServiceHost.Abort();
}
But _ESSServiceHost.Abort() should be called internally by the Close method if needed.
Target framework is 4.5, installed is .NET 4.7.2.
Found out that probably the service hangs after series of malformed requests. Expected record type 'Version', found '71'. etc.
I have found in the svclog file that my service hangs after series of malformed request that happen on Saturday and Sunday at approx. 5:15 AM. The error messages were Expected record type 'Version', found '71'., Error while reading message framing format at position 0 of stream (state: ReadingVersionRecord). But I could not find the cause of theese malformed request series, so I tried to fix the service to withstand the "attack".
I have modified the OnStop method as follows:
protected override void OnStop()
{
try
{
if (_ESSServiceHost != null)
{
Tools.LogInfo("Stopping ESService.");
var abortTask = Task.Run(() => _ESSServiceHost.Abort());
var closeTask = Task.Run(() => _ESSServiceHost.Close(TimeSpan.FromSeconds(300)));
try
{
if (_ESSServiceHost.State == CommunicationState.Faulted)
{
Tools.LogInfo("ESSServiceHost.State == CommunicationState.Faulted");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
else
{
if (!closeTask.Wait(TimeSpan.FromSeconds(301)))
{
Tools.LogInfo("Failed to Close - trying Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
}
}
catch (Exception ex)
{
Tools.LogException(ex, "ESSServiceHost.Close");
try
{
Tools.LogInfo("Abort.");
if (!abortTask.Wait(TimeSpan.FromSeconds(60)))
Tools.LogInfo("Failed to Abort.");
}
catch (Exception ex2)
{
Tools.LogException(ex2, "ESSServiceHost.Abort");
}
}
_ESSServiceHost = null;
Tools.LogInfo("ESService stopped.");
}
}
catch (Exception ex)
{
Tools.LogException(ex,"OnStop");
}
}
Today on Monday I have checked the svclog and the "attacks" with malformed request remained there but my service lived happily through it. So it seemed to be fixed. Moreover only:
Stopping ESService.
ESService stopped.
events were logged in my log file. No aborts etc. So I guess that putting the Close call on a separate thread fixed the problem but absolutely do not know why.

System.Threading.Task not starting in production environment

I have a c# windows service which is doing various tasks. Its working perfectly on my local system but as soon as I start it on my product server, its doesn't perform a particular task on it.
This is how my service is structured:
public static void Execute()
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
string cc = dr["ccode"].ToString();
Task objTask = new Task(delegate { RequestForEachCustomer(cc); });
objTask.Start();
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestForEachCustomer(object cc)
{
try
{
// .... some work ....
foreach (DataRow dr in dt.Rows)
{
WriteLog("RequestForEachCustomer - Before Task");
Task objTask = new Task(delegate { RequestProcessing(dr); });
objTask.Start();
WriteLog("RequestForEachCustomer - After Task");
}
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
public static void RequestProcessing(object dr)
{
try
{
WriteLog("Inside RequestProcessing");
// .... some work ....
}
catch (Exception ex)
{
// Logging in DB + Text File
}
}
Now what happens on the production server is that it logs the last entry in RequestForEachCustomer which is "RequestForEachCustomer - After Task" but it doesn't log the entry from RequestProcessing which mean the task is not starting at all. There are no exceptions in either database or text file.
There are no events logged in window's event viewer either. Also the service keeps working (if I insert another record in database, its processed by the service immediately so the service isn't stuck either. It just doesn't seem to process RequestProcessing task.)
I am baffled by this and it would be great if someone could point out the mistake I am making. Oh, btw did I forgot to mention that this service was working perfectly few days ago on the server and it is still working fine on my local PC.
EDIT :
WriteLog :
public static void WriteErrorLog(string Message)
{
StreamWriter sw = null;
try
{
lock (locker)
{
sw = new StreamWriter(AppDomain.CurrentDomain.BaseDirectory + "\\Logs\\LogFile.txt", true);
sw.WriteLine(DateTime.Now.ToString() + ": " + Message);
sw.Flush();
sw.Close();
}
}
catch (Exception excep)
{
try
{
// .... Inserting ErrorLog in DB ....
}
catch
{
throw excep;
}
throw excep;
}
}
I have also logged an entry on OnStop() something like "Service Stopped" and its logs every time I stop my service so the problem couldn't exist in WriteLog function.
I suggest you refactor your code as in this MSDN example. What bother me in your code is, you never wait for tasks to finish anywhere.
The following example starts 10 tasks, each of which is passed an index as a state object. Tasks with an index from two to five throw exceptions. The call to the WaitAll method wraps all exceptions in an AggregateException object and propagates it to the calling thread.
Source : Task.WaitAll Method (Task[])
This line from example might be of some importance :
Task.WaitAll(tasks.ToArray());

Try inside catch to ensure finally executes

I have to process items off a queue.
Deleting items off the queue is a manual call to Queue.DeleteMessage. This needs to occurs regardless of whether or not the processing succeeds.
var queueMessage = Queue.GetMessage();
try
{
pipeline.Process(queueMessage);
}
catch (Exception ex)
{
try
{
Logger.LogException(ex);
}
catch { }
}
finally
{
Queue.DeleteMessage(queueMessage);
}
Problem:
On failure, I log the error to some data store. If this logging fails (perhaps the data store is not available), I still need the message to be deleted from the queue.
I have wrapped the LogException call in another try catch. Is this the correct way or performing thing?
Following code is enough. finally blocks execute even when exception is thrown in catch block.
var queueMessage = Queue.GetMessage();
try
{
pipeline.Process(queueMessage);
}
catch (Exception ex)
{
Logger.LogException(ex);
}
finally
{
Queue.DeleteMessage(queueMessage);//Will be executed for sure*
}
The finally block always executes, even if it throws an unhandled error (unless it end the app). So yes.

Handling Error "WebDev.WebServer.Exe has stopped working"

Is there a way to handle the error "WebDev.WebServer.Exe has stopped working" in ASP.NET and keep the page running or even the just the WebServer running? Or is this an impossible task and is essentially like asking how to save someone's life after they've died?
I have the error-causing code inside a try/catch block, but that doesn't make a difference. I've also tried registering a new UnhandledExceptionEventHandler, but that didn't work either. My code is below in case I'm doing something wrong.
Also to be clear, I'm not asking for help on how to prevent the error; I want to know if and when the error happens if there's anything I can do to handle it.
UPDATE 1: TestOcx is a VB6 OCX that passes a reference of a string to a DLL written in Clarion.
UPDATE 2: As per #JDennis's answer, I should clarify that the catch(Exception ex) block is not being entered either. If I removed the call to the OCX from the try\catch block it still won't reach the UnhandledException method. There are essentially two areas that don't ever get executed.
UPDATE 3: From #AndrewLewis, I tried to also add a regular catch block to catch any non-CLS compliant exceptions, and this did not work either. However, I later found that since .NET 2.0 on, all non-CLS exceptions are wrapped inside RuntimeWrappedException so a catch (Exception) will catch non-CLS compliant exceptions too. Check out this other question here for more info.
public bool TestMethod()
{
AppDomain.CurrentDomain.UnhandledException +=
new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
string input = "test";
string result = "";
try
{
TestOcx myCom = new TestOcx();
result = myCom.PassString(ref input); // <== MAJOR ERROR!
// do stuff with result...
return true;
}
catch (Exception ex)
{
log.Add("Exception: " + ex.Message); // THIS NEVER GETS CALLED
return false;
}
}
private static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
// THIS NEVER GETS CALLED
try
{
Exception ex = (Exception)e.ExceptionObject;
log.Add("Exception: " + ex.Message);
}
catch (Exception exc)
{
log.Add("Fatal Non-UI Error: " + exc.Message);
}
}
You should try catching non-CLS compliant exceptions to make sure nothing is being thrown (keep in mind you don't want to do this in production, always be specific!):
try
{
TestOcx myCom = new TestOcx();
result = myCom.PassString(ref input); // <== MAJOR ERROR!
// do stuff with result...
return true;
}
catch (Exception ex)
{
log.Add("Exception: " + ex.Message); // THIS NEVER GETS CALLED
return false;
}
catch
{
//do something here
}
Your code reads //THIS NEVER GETS CALLED.
If you catch the exception it is no longer un-handled. this is why it doesn't fire an unhandledexception event.

singleton messagebox

I am writing an application in which when something happened to the connection I want to pop up a messagebox and show the user the error...
for this purpose when the program throw an exception it will come to the catch block and in that I want to show the user the message here is the code :
catch (WebException ex)
{
if (!(ex.Message == "The operation has timed out."))
{
MessageBox.Show(ex.Message);
}
}
As it seems the program will come to this catch something like forever till the connection is become fixed so what should I do to update my message on just one messagebox at a time?
There is not much control over MessageBox when it's displayed. I would use another Form displayed in a modal mode. Before displaying, you can start a separate thread and put the logic to monitor the connection. When re-established, notify the message form and close it.
You can use something like:
public static class FailureMessagebox
{
private static bool _hasBeenSuccessful = true;
public static void ShowIfFailure(Action someAction)
{
try
{
someAction();
_hasBeenSuccessful = false;
}
catch (Exception err)
{
if (_hasBeenSuccessful)
MessageBox.Show(ex.Message);
_hasBeenSuccessful = false;
throw;
}
}
}
Sample usage:
try
{
WebResponse response;
FailureMessagebox.ShowIfFailure(() => response = webRequest.GetResponse());
}
catch (WebException err)
{
//handle the exception here.
}

Categories