Azure WebJob Command Timeout - c#

We are having issue with Azure Web Jobs. We created a C# console application, zipped it, and created the new Web Job. It's a c# console app that will constantly hit one of our web services to process items in queue.
Whenever we run the Web Job, we are getting the following error:
'cmd /c xxxxxxxx....' aborted due to no output and CPU activity for
121 seconds. You may increase SCM_COMMAND_IDLE_TIMEOUT setting to
solve the issue
When we increased the SCM_COMMAND_IDLE_TIMEOUT to 600 (10 minutes). The job DOES run for 10 minutes - and then we get the same error with the same 121 seconds error.
What are we doing wrong?
Here is the console app code:
static void Main(string[] args)
{
bool ThereAreItemsInQueue = true;
int Counter = 1;
DateTime StartTime = DateTime.Now;
while(ThereAreItemsInQueue)
{
Task.Run(() => {
try
{
//DEQUEUE
byte[] response = HttpHelper.HttpPOST(#"xxxxxxxxxxxxx", new byte[0]);
string strResponse = System.Text.Encoding.Default.GetString(response);
System.Diagnostics.Trace.TraceError("Attempt #" + Counter + "DEQUEUE FINISHED. Response:" + strResponse);
//CHECK IF THE QUEUE IS EMPTY
if (strResponse.Contains("Were Done"))
ThereAreItemsInQueue = false;
}
catch(Exception ex)
{
System.Diagnostics.Trace.TraceError("Error Has Occured on attempt #" + Counter + "." + ex.Message + "\r" + ex.StackTrace);
}
});
System.Threading.Thread.Sleep(5000);
//SEE IF THIS HAS BEEN RUNNING FOR MORE THAN 24 HOURS
if (DateTime.Now.Subtract(StartTime).TotalHours >= 24)
ThereAreItemsInQueue = false;
Counter++;
}
}
Are we approaching this problem the wrong way?
Note: each HttpHelper.HttpPOST request takes about 2 seconds - so that's not the issue.
Note2: We are using Task.Run to create "set-it-and-forget-it" type of requests.
Note3: The website setting of "Always On" - is turned on.

For triggered WebJobs the way to increase idle timeout is using the app setting: WEBJOBS_IDLE_TIMEOUT. Set it to your desired timeout in seconds.
The error is confusing and only refers to idle timeout during deployment.
https://github.com/projectkudu/kudu/wiki/Web-jobs#configuration-settings

This seems to have solved my problem:
if (Counter % 25 == 0)
Console.WriteLine("Heartbeat");
I guess you have to keep writing out to console to keep the JOB running.

Related

Why does discord.net sometimes run multiple times through code with a single instance?

I am running a discord bot which hosts a game 24/7. Sometimes the bot runs some code double while it should only run it just once, which in my case causes a crash because it uses the same datareader at the same time. Problem is, I have looked through my code multiple times, and with multiple sets of eyes, but we cannot find a clue why it runs it double since it only should run it once. Some facts are:
There is only one instance of the program.
The code that is run twice is only called once in code.
When the bot is restarted it is all fine, but after some time it happens again.
My feeling is that it has something to do with disconnecting and reconnecting to the discord servers. Because it mostly (maybe only) happens when the bot is idle for some time. I noticed that the bot periodically sends a 'Ready' message in the logchat, which it only does when it is restarted. But I cannot find a reason why, or how to fix it.
I have an example of what happens on discord, followed by a crash:
> MyBot BOT — 07/17/2021
> Not enough players (6/10) to start a game, trying again tomorrow 15:00.
> MyBot BOT — 07/18/2021
> #Inactief, Not enough players (6/10) to start a game, trying again at 15:30.
> #Inactief, Not enough players (6/10) to start a game, trying again at 15:30.
> #Inactief, Not enough players (6/10) to start a game, trying again at 15:30.
The last sentence is ran three times while it should only mention this once. This means that while there is only one instance running there seems to be going something wrong which means this method is called three times.
public async Task StartNewGame(string type, bool forced, bool speed, List<ulong> players = null)
{
int playercount;
if (players == null)
{
// here it gets the players, which works fine
}
else
{
playercount = players.Count;
}
if (playercount >= 10 || forced)
{
//some code
}
else
{
DateTime newtry;
if (DateTime.Now.AddMinutes(30).Hour < 20 || (DateTime.Now.AddMinutes(30).Hour == 20 && DateTime.Now.AddMinutes(30).Minute == 0))
{
newtry = DateTime.Now.AddMinutes(30);
if (DateTime.Now.Hour == 15 && DateTime.Now.Minute < 30)
await (_bot.server.GetChannel(_bot.gebeurtenischat.Id) as IMessageChannel).SendMessageAsync(_bot.Inactief.Mention + ", Not enough players (" + (playercount) + "/10) to start a game, trying again at " + newtry.ToString("HH:mm") + ".");
else
await (_bot.server.GetChannel(_bot.gebeurtenischat.Id) as IMessageChannel).SendMessageAsync("Not enough players (" + (playercount) + "/10) to start a game, trying again at " + newtry.ToString("HH:mm") + ".");
}
else
{
newtry = new DateTime(DateTime.Now.Year, DateTime.Now.Month, DateTime.Now.Day).AddHours(39);
await (_bot.server.GetChannel(_bot.gebeurtenischat.Id) as IMessageChannel).SendMessageAsync("Not enough players (" + (playercount) + "/10) to start a game, trying again tomorrow " + newtry.ToString("HH:mm") + ".");
}
cmdText = string.Format("update servers set deadline = '{0}' where serverid = '{1}'", newtry.ToString("yyyy-MM-dd HH:mm:ss"), _bot.server.Id);
MySqlCommand cmd = new MySqlCommand(cmdText, conn);
cmd.ExecuteNonQuery();
await GetSpel();
}
}
The only place where this code is called is at this piece of code:
DateTime deadline = Convert.ToDateTime(dt.Rows[0][0]);
int ms = (int)(deadline - DateTime.Now).TotalMilliseconds;
if (ms > 0)
{
Console.WriteLine("Waiting until " + deadline.ToLongTimeString() + " to start a new
game.");
await Task.Delay(ms);
}
await StartNewGame("All Any", false, false);
This is where the bot will idle when it is (re)started and waiting to start a new game. But it starts with one time going through, and after a night of sleep it happens to run twice or thrice at the same time through the code.
Has anyone experienced this, or does anyone know where to start looking for a solution?
P.S. Please let me know if you need any more details. This issue has been bugging me for months and I can't seem to figure it out.

C# winforms freezing: Serialport, Timer, Thread

Edit: Keeping the original question for continuity.
I then edited the question with replacement code for the ReadLine() method by using ReadExisting instead. It works however I still have the same freeze, where the app becomes unresponsive. Debug says it's locking (it takes a while to freeze, sometimes seconds, sometimes minutes) in the while () {} function where I wait for the complete message. More explanations below:
-- obsolete --
What is a good way to handle serialport.readtimeout exception?
try
{
serialPort1.Write(Command_);
if (!IsWriteComm_)
{
Response_ = serialPort1.ReadLine().Replace("\r", "");
}
}
catch (TimeoutException err)
{
DateTime d = DateTime.Now;
rtboxDiag.AppendText("\n" + d.ToString("HH:mm:ss") + ": ");
rtboxDiag.AppendText(err.Message);
if (!serialPort1.IsOpen)
InitConnection();
return Textbox_;
}
this bit of code is exectuted on a timer tick event.
I was having a weird "crash" of the app with an IO exception
"The I/O operation has been aborted because of either a thread exit or an application request."
no matter what I do I am not able to "recover" meaning, I am no longer able to poll data from the serial port.
I added this exception catch and it does log the exception. weirdly enough the test on !serialport.isopen is false (meaning the port is still open).
What might be a hint is: this error does STOP the timer somehow, this is not something I am doing in code. so I am suspecting something related to the timer, rather than the serialport, but I could be wrong.
Closing the port manually, and reconnecting does not fix the problem.
Disconnecting and reconnecting the USB does not fix the problem.
however, closing the app, and relaunching the app does fix the problem (without even disconnecting the MCU or power cycling the MCU/hardware).
-- /obsolete --
edit: the problem is appearing after a few seconds, sometimes minutes of flawless operations. I cannot repeat the issue using a serialport terminal polling the data the same way, at the same frequency. It seems the problem is not coming from the hardware itself.
cheers
Edit: I have yet to test the following modification, not sure if it will fix this problem (I doubt), but at least it's an attempt at not using .readline() which from what I've gathered is not good practice.
anyway here it is:
try
{
serialPort1.Write(Command_);
if (!IsWriteComm_)
{
while (!SerialRxCplt) ;
Response_ = SerialRxResponse.Replace("\r", "").Replace("\n", "");
SerialRxCplt = false;
//Response_ = serialPort1.ReadLine().Replace("\r", "");
}
}
catch (TimeoutException err)
{
DateTime d = DateTime.Now;
rtboxDiag.AppendText("\n" + d.ToString("HH:mm:ss") + ": ");
rtboxDiag.AppendText(err.Message);
if (!serialPort1.IsOpen)
InitConnection();
return Textbox_;
}
and I have the datareceived event enabled:
private void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
var serialPort = (System.IO.Ports.SerialPort)sender;
string dataReceived = serialPort.ReadExisting();
ProcessSerialData(dataReceived);
}
and this is how I am processing the data, and manually "waiting" for the \n character which tells me when the data has been fully received.
private void ProcessSerialData(string data)
{
SerialRxBuffer += data;
if (SerialRxBuffer.Contains("\n"))
{
SerialRxCplt = true;
SerialRxResponse = SerialRxBuffer;
SerialRxBuffer = "";
}
else
{
SerialRxCplt = false;
}
}
any input is welcome.
I have added "stuff" for debugging inside that while loop and it does work fine for a while and then freezes, no error or exception is thrown there. For some reason I have a feeling it's not related to the serial port.
I have even added this:
try
{
serialPort1.Write(Command_);
if (!IsWriteComm_)
{
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
while (!SerialRxCplt || Timer2StopWatchMilli > 5)
{
Timer2StopWatchMilli = stopWatch.Elapsed.TotalMilliseconds;
ExceptionMessage = Timer2StopWatchMilli.ToString();
IsException = true;
}
stopWatch.Stop();
if (!SerialRxCplt)
return Textbox_;
Response_ = SerialRxResponse.Replace("\r", "").Replace("\n", "");
SerialRxCplt = false;
//Response_ = serialPort1.ReadLine().Replace("\r", "");
}
}
the ExceptionMessage and IsException help me have an idea of what's happening in that loop. And in normal operations, it is what you would except, increments in the order of 0.0x milliseconds. Data is being processed correctly. When it freezes, nothing looks abnormal. I initially thought I was somehow getting "stuck" in an infinite loop but that || Timer2StopWatchMilli > 5 should get me out of it, acting as some sort of timeout.
one extra piece of info: when it freezes, the one CPU core is fully loaded. (I have a 6core CPU, and it's 16-17% in the task manager - memory usage is low < 30MB)
Any help is welcome
I fixed it by clearing RX/TX and stream buffers after each successful transaction.
I think data was being sent to the PC faster than it was able to read causing data to eventually accumulating on the Rx Buffer.
private void SerialPortClearBuffers()
{
serialPort1.DiscardOutBuffer();
serialPort1.DiscardInBuffer();
serialPort1.BaseStream.Flush();
}

Com object method is blocking in asp.net core but not unit tests

I have a class I have used to wrap up a com object (MLApp) that is used to call into matlab. I have a unit test that initializes 5 matlab sessions and runs a command that pauses for 10 seconds in each individual session. My unit test takes ~10 seconds to run which is expected since they should be executing in parallel.
[Test]
public void ParallelExecution()
{
List<MatlabRunner> runners = new List<MatlabRunner>();
for (int i = 0; i < 5; i++)
{
runners.Add(new MatlabRunner());
}
try
{
List<Task<string>> t = new List<Task<string>>();
foreach (MatlabRunner r in runners)
t.Add(Task.Run<string>(()=>r.Execute( "pause(10)" )));
Task<string>.WaitAll(t.ToArray());
t.ForEach((task) => System.Console.WriteLine(task.Result));
}
finally
{
runners.ForEach((r)=>r.Quit());
}
}
However, now when I put what amounts to the same code in my asp.net core project, I can make two web requests and they are both received in parallel (I know this because I log that the request is received) but then the call into runner.Execute() is serialized.
[HttpGet("Verify")]
public JsonResult Verify(String lot = null,String lotStatus = null,int? NumPorts = null,int? wafer = null)
{
System.IO.File.AppendAllText(#"C:\Logs\log.txt", "\n\Received Request! " + System.DateTime.Now + "\n");
String command = "pause(10)";
MatlabRunner runner = null;
String result = "";
try
{
runner = new MatlabRunner();
result = runner.Execute(command);
}
catch(Exception e)
{
System.IO.File.AppendAllText(#"C:\Logs\log.txt", e.StackTrace + " " + e.InnerException + " " + e.GetType() + e.Source + " " + e.GetBaseException());
}
finally
{
runner.Quit();
System.IO.File.AppendAllText(#"C:\Logs\log.txt", "\nQuit! " + System.DateTime.Now + "\n");
}
return new JsonResult(result);
}
The problem is that I make two, parallel GET requests to this Verify method. While one request is calling runner.Execute("pause(10)") the other request is blocked from calling this method. I would expect to make two calls to this Verify method in parallel and them both to execute in parallel, finishing in a total of ~10 seconds. However, for both requests to complete takes 20 seconds (10 for the first, 10 for the second). In my unit test, I call this same method in parallel using Tasks and the entire unit test executes in 10 seconds, despite having 5 runners launched. If I were to do the same in the web service, it would take 50 seconds. Hopefully this is enough info. I can attach a log with time stamps to show how things are executing, if it would be valuable.
Am I missing something from a configuration perspective in IIS? Potentially something when I build or publish?

Increasing a timer everytime an evaluation is false

I'm measuring the amount of time certain actions in Internet Explorer take. One of the actions I'm measuring is launching another application via a link in IE. To measure how long this application takes to launch I start a timer after the link has been clicked in IE and I had planned on stopping it once the application had fully loaded. The problem there was that my application would go to execute the next line since the evaluation would always be false as there wasn't enough time between lines for the external app to load.
try
{
Process[] externApp = Process.GetProcessesByName("External");
System.Timers.Timer runningWindow = new System.Timers.Timer(1000);
runningWindow.Start();
while (runningWindow.Enabled)
{
if (externApp[0].Responding)
{
timer.Stop();
output[2] = timer.Elapsed.Seconds.ToString();
runningWindow.Stop();
}
runningWindow.Interval += 100;
}
externApp[0].Kill();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
Above is how I'm currently trying to get my application to hang while I wait for the other application to respond.
Since I wasn't sure how to wait for another application to load I decided to use a timer to postpone the execution of the next line after I check if the other application is running. If the application wasn't running I would increase the timer by a tenth of a second then check again. However the problem I was running into here is that I don't think I'm increasing the timer as my time measurement is always 0 for this step.
So how do I increase my timer if my external application isn't responding?
Albeit unreliable for the reasons explained above in comments you can still use the Process.WaitForInputIdle to get an approximate evaluation of the startup time required by your app
// Launch the external app...
ProcessStartInfo psi = new ProcessStartInfo();
psi.FileName = #"D:\temp\MyExternalApp.exe";
psi.WorkingDirectory = #"D:\temp";
// Init measure
Stopwatch sw = StopWatch.StartNew();
Process.Start(psi);
Process[] externApp = null;
int cnt = 0;
bool ready = false;
while (externApp == null || externApp.Length == 0 || cnt == 600)
{
// Loop until the process appears in the process list.
// This is taxing for the performances. Heisenberg here
externApp = Process.GetProcessesByName("MyExternalApp");
Thread.Sleep(100);
cnt++;
}
if(cnt >= 600)
MessageBox.Show("Something has gone terribly wrong launching the external app");
else
ready = externApp[0].WaitForInputIdle(30000);
sw.Stop();
if(!ready)
MessageBox.Show("Not ready after:" + sw.ElapsedMilliseconds + " ms");
else
MessageBox.Show("Ready after:" + sw.ElapsedMilliseconds + " ms");
Another approch could be used (still with WaitForInputIdle) if you cannot start a Stopwatch the exact instant in which your external process has been started (like it seems with your comment about starting the ExternalApp through a browseer) In this case we could try to get the start time using the same named property in the Process class
Process[] externApp = null;
int cnt = 0;
bool ready = false;
while (externApp == null || externApp.Length == 0 || cnt == 600)
{
// Again we are waiting to see the process in the list of processes
externApp = Process.GetProcessesByName("MyExternalApp");
Thread.Sleep(100);
cnt++;
}
if(cnt >= 600)
MessageBox.Show("Something has gone terribly wrong launching the external app");
else
{
ready = externApp[0].WaitForInputIdle(30000);
DateTime readyAt = DateTime.Now;
TimeSpan ts = readyAt - externApp[0].StartTime;
MessageBox.Show("Ready after:" + ts.TotalMilliseconds + " ms");
}
You can simply get the Current time before starting IE:
DateTime start = DateTime.Now;
And in the end subtract it from the finish time like:
double SecondsElapsed = (DateTime.Now-start).TotalSeconds;

Replace Thread.Sleep with System.Threading.Timer?

I'm trying to replace Thread.Sleep with System.Threading.Timer, and I'm trying to implement it in the following code. The main reason for changing is that, even though it works when testing locally, it's not working properly in the server. Plus, I've read that using it here is bad practice.
I've seen several examples (including the one below), but I'm not certain how I could use it in my case: System.Threading.Timer in C# it seems to be not working. It runs very fast every 3 second
In my console app, I need to copy files to our server every 15 minutes. So at :20, :35, :50, :05, I begin reading files for that quarter. In the case below, the files will be available at :45, and I add 5 minutes just in case.
This is my code. I had previously tried to copy files from several quarters in parallel, but the server where the source files reside is having trouble with that. So I'm going back to this.
My question is, how can I replace Thread.Sleep with System.Threading.Timer in this example?
I wanted to try await Task.Delay(Int32), but I have VS2010:
DateTime lastTimeRead = new DateTime(2014, 9, 9, 8, 35, 0); //Last read at 8:35AM
DateTime nextTimeRead;
for (; ; )
{
now = DateTime.Now; //It's currently 8:43AM
nextTimeRead = LastTimeRead.AddMinutes(15); // nextTimeRead = 8:50AM.
if (nextTimeRead > now) //Yes, so wait 7 minutes for files to be available
{
TimeSpan span = nextTimeRead.Subtract(now);
Double milliseconds = span.TotalMilliseconds;
Console.WriteLine("Sleep for milliseconds: " + milliseconds.ToString());
Thread.Sleep(Convert.ToInt32(milliseconds));
Console.WriteLine("Download files after sleep of: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}
else // Files are available. Read.
{
Console.WriteLine("Download files no sleep: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}
LastTimeRead = nextTimeRead;
}
The idea is to have a timer and Enable it when you want to set the delay in your program:
System.Timers.Timer Delay = new System.Timers.Timer();
Delay.Elapsed += new System.Timers.ElapsedEventHandler(Delay_Elapsed);
Delay.Interval=Convert.ToInt32(milliseconds);
Delay.Enabled = false;
void Delay_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
Delay.Enabled = false;
}
......
.....
if (nextTimeRead > now) //Yes, so wait 7 minutes for files to be available
{
TimeSpan span = nextTimeRead.Subtract(now);
Double milliseconds = span.TotalMilliseconds;
Console.WriteLine("Sleep for milliseconds: " + milliseconds.ToString());
Delay.Enabled = true;
while (Delay.Enabled)
{
////Wait until time passes
}
Console.WriteLine("Download files after sleep of: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}

Categories