We Keep Coding

Why does calling the Tesseract process cause this service to crash randomly?

I have a .NET Core 2.1 service which runs on an Ubuntu 18.04 VM and calls Tesseract OCR 4.00 via a Process instance. I would like to use an API wrapper, but I could only find one available and it is only in beta for the latest version of Tesseract -- the stable wrapper uses version 3 instead of 4. In the past, this service worked well enough, but I have been changing it so that document/image data is written and read from disk less frequently in an attempt to improve speed. The service used to call many more external processes (such as ImageMagick) which were unnecessary due to the presence of an API, so I have been replacing those with API calls.
Recently I've been testing this with a sample file taken from real data. It's a faxed document PDF that has 133 pages, but is only 5.8 MB in spite of that due to grayscale and resolution. The service takes a document, splits it into individual pages, then assigns multiple threads (one thread per page) to call Tesseract and process them using Parallel.For. The thread limits are configurable. I am aware that Tesseract has its own multithreading environment variable (OMP_THREAD_LIMIT). I found in prior testing that setting it to "1" is ideal for our set up at the moment, but in my recent testing for this issue I have tried leaving it unset (dynamic value) with no improvement.
The issue is that unpredictably, when Tesseract is called, the service will hang for about a minute and then crash, with the only error showing in journalctl being:
dotnet[32328]: Error while reaping child. errno = 10
dotnet[32328]: at System.Environment.FailFast(System.String, System.Exception)
dotnet[32328]: at System.Environment.FailFast(System.String)
dotnet[32328]: at System.Diagnostics.ProcessWaitState.TryReapChild()
dotnet[32328]: at System.Diagnostics.ProcessWaitState.CheckChildren(Boolean)
dotnet[32328]: at System.Diagnostics.Process.OnSigChild(Boolean)
I can't find anything at all online for this particular error. It would seem to me, based on related research I've done on the Process class, that this is occurring when the process is exiting and dotnet is trying to clean up the resources it was using. I'm really at a loss as to how to even approach this problem, although I have tried a number of "guesses" such as changing thread limit values. There is no cross-over between threads. Each thread has its own partition of pages (based on how Parallel.For partitions a collection) and it sets to work on those pages, one at a time.
Here is the process call, called from within multiple threads (8 is the limit we normally set):
private bool ProcessOcrPage(IMagickImage page, int pageNumber, object instanceId)
{
var inputPageImagePath = Path.Combine(_fileOps.GetThreadWorkingDirectory(instanceId), $"ocrIn_{pageNumber}.{page.Format.ToString().ToLower()}");
string outputPageFilePathWithoutExt = Path.Combine(_fileOps.GetThreadOutputDirectory(instanceId),
$"pg_{pageNumber.ToString().PadLeft(3, '0')}");
page.Write(inputPageImagePath);
var cmdArgs = $"-l eng \"{inputPageImagePath}\" \"{outputPageFilePathWithoutExt}\" pdf";
bool success;
_logger.LogStatement($"[Thread {instanceId}] Executing the following command:{Environment.NewLine}tesseract {cmdArgs}", LogLevel.Debug);
var psi = new ProcessStartInfo("tesseract", cmdArgs)
{
RedirectStandardError = true,
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
};
// 0 is not the default value for this environment variable. It should remain unset if there
// is no config value, as it is determined dynamically by default within OpenMP.
if (_processorConfig.TesseractThreadLimit > 0)
psi.EnvironmentVariables.Add("OMP_THREAD_LIMIT", _processorConfig.TesseractThreadLimit.ToString());
using (var p = new Process() { StartInfo = psi })
{
string standardErr, standardOut;
int exitCode;
p.Start();
standardOut = p.StandardOutput.ReadToEnd();
standardErr = p.StandardError.ReadToEnd();
p.WaitForExit();
exitCode = p.ExitCode;
if (!string.IsNullOrEmpty(standardOut))
_logger.LogStatement($"Tesseract stdOut:\n{standardOut}", LogLevel.Debug, nameof(ProcessOcrPage));
if (!string.IsNullOrEmpty(standardErr))
_logger.LogStatement($"Tesseract stdErr:\n{standardErr}", LogLevel.Debug, nameof(ProcessOcrPage));
success = p.ExitCode == 0;
}
return success;
}
EDIT 4: After much testing and discussion with Clint in chat, here is what we learned. The error is raised from a Process event "OnSigChild," that much is obvious from the stack trace, but there is no way to hook into the same event that raises this error. The process never times out given a timeout of 10 seconds (Tesseract typically only takes a few seconds to process a given page). Curiously, if the process timeout is removed and I wait on the standard output and error streams to close, it will hang for a good 20-30 seconds, but the process does not appear in ps auxf during this hang time. From the best that I can tell, Linux is able to determine that the process is done executing, but .NET is not. Otherwise, the error seems to be raised at the very moment that the process is done executing.
The most baffling thing to me is still that the process handling part of the code really hasn't changed very much compared to the working version of this code we have in production. This suggests that it's an error I made somewhere, but I am simply unable to find it. I think I will have to open up an issue on the dotnet GitHub tracker.

"Error while reaping child"
Processes hold up some resources in the kernel, On Unix, when the parent dies, it is the init process that is responsible for cleaning up the kernel resources both Zombine and Orphan process (aka reaping the child). .NET Core reaps child processes as soon as they terminate.
"I have discovered that removing the stdout and stderr stream ReadToEnd
calls causes the processes to end immediately instead of hang, with
the same error"
The error is due to the fact that you are prematurely calling p.ExitCode even before the process has finished and with the ReadToEnd you are just delaying this activity
Summary of updated code
StartInfo.FileName should point to a filename that you want to start
UseShellExecute to false if the process should be created directly from the executable file and true if you intend that shell should be used when starting the process;
Added asynchrnous read operations to standard ouput and error streams
AutoResetEvents to signal when the output and error when the operations complete
Process.Close() to release the resources
It is easier to set and use ArgumentList over Arguments property
Redhat Blog on NetProcess on Linux
Revised Module
private bool ProcessOcrPage(IMagickImage page, int pageNumber, object instanceId)
{
StringBuilder output = new StringBuilder();
StringBuilder error = new StringBuilder();
int exitCode;
var inputPageImagePath = Path.Combine(_fileOps.GetThreadWorkingDirectory(instanceId), $"ocrIn_{pageNumber}.{page.Format.ToString().ToLower()}");
string outputPageFilePathWithoutExt = Path.Combine(_fileOps.GetThreadOutputDirectory(instanceId),
$"pg_{pageNumber.ToString().PadLeft(3, '0')}");
page.Write(inputPageImagePath);
var cmdArgs = $"-l eng \"{inputPageImagePath}\" \"{outputPageFilePathWithoutExt}\" pdf";
bool success;
_logger.LogStatement($"[Thread {instanceId}] Executing the following command:{Environment.NewLine}tesseract {cmdArgs}", LogLevel.Debug);
using (var outputWaitHandle = new AutoResetEvent(false))
using (var errorWaitHandle = new AutoResetEvent(false))
{
try
{
using (var process = new Process())
{
process.StartInfo = new ProcessStartInfo
{
WindowStyle = ProcessWindowStyle.Hidden,
FileName = "tesseract.exe", // Verify if this is indeed the process that you want to start ?
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = true,
Arguments = cmdArgs,
WorkingDirectory = Path.GetDirectoryName(path)
};
if (_processorConfig.TesseractThreadLimit > 0)
process.StartInfo.EnvironmentVariables.Add("OMP_THREAD_LIMIT", _processorConfig.TesseractThreadLimit.ToString());
process.OutputDataReceived += (sender, e) =>
{
if (e.Data == null)
{
outputWaitHandle.Set();
}
else
{
output.AppendLine(e.Data);
}
};
process.ErrorDataReceived += (sender, e) =>
{
if (e.Data == null)
{
errorWaitHandle.Set();
}
else
{
error.AppendLine(e.Data);
}
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
if (!outputWaitHandle.WaitOne(ProcessTimeOutMiliseconds) && !errorWaitHandle.WaitOne(ProcessTimeOutMiliseconds) && !process.WaitForExit(ProcessTimeOutMiliseconds))
{
//To cancel the read operation if the process is stil reading after the timeout this will prevent ObjectDisposeException
process.CancelOutputRead();
process.CancelErrorRead();
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("Timed Out");
//To release allocated resource for the Process
process.Close();
//Timed out
return false;
}
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("Completed On Time");
exitCode = process.ExitCode;
if (!string.IsNullOrEmpty(standardOut))
_logger.LogStatement($"Tesseract stdOut:\n{standardOut}", LogLevel.Debug, nameof(ProcessOcrPage));
if (!string.IsNullOrEmpty(standardErr))
_logger.LogStatement($"Tesseract stdErr:\n{standardErr}", LogLevel.Debug, nameof(ProcessOcrPage));
process.Close();
return exitCode == 0 ? true : false;
}
}
Catch
{
//Handle Exception
}
}
}

Communicate with process in C#

I need to communicate with external executable (ampl.exe) using standard input and standard output. This exe make calculations during some minutes with some display in the console. It has a prompt so I can succesively launch calculations by using its standard input as soon as a calculation is finished.
The external exe is launched as :
var myProcess = new Process();
myProcess.StartInfo = new ProcessStartInfo("ampl.exe");
myProcess.StartInfo.CreateNoWindow = true;
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.RedirectStandardError = true;
myProcess.StartInfo.RedirectStandardInput = true;
myProcess.Start();
I communicate with it by using myProcess.StandardInput and myProcess.StandardOutput (synchronous way).
I use standard input to launch the calcul, for example :
myProcess.StandardInput.WriteLine("solve;");
I want to wait the end of the solve statement, get results in files, prepare new calculation input files and then launching a second solve.
My problem is that I do now know when the first calculation is finished, that is when the exe is waiting for new command in its standard input.
The only way I found is to add a specific display command and wait for getting it it its standard output :
myProcess.StandardInput.WriteLine("solve;");
myProcess.StandardInput.WriteLine("print 'calculDone';");
string output = myProcess.StandardOutput.ReadLine();
while (!output.Contains("calculDone"))
{
output = myProcess.StandardOutput.ReadLine();
}
Is there another way avoiding to use this display command to do this ?
Edit : following advices, I tried the asynchronous way. But I still need to print 'CalculDone' to know when the solve statement ended. I do not get the prompt of ampl.exe (which is 'ampl : ') in the standard output of the process.
AutoResetEvent eventEnd = new AutoResetEvent(false);
var myProcess = new Process();
myProcess.StartInfo = new ProcessStartInfo("ampl.exe");
myProcess.StartInfo.CreateNoWindow = true;
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.RedirectStandardError = true;
myProcess.StartInfo.RedirectStandardInput = true;
myProcess.EnableRaisingEvents = true;
myProcess.OutputDataReceived += (sender, e) =>
{
if (e.Data == "commandDone")
{
eventEnd.Set();
}
else if (e.Data != null)
{
Console.WriteLine("ampl: {0}", e.Data);
}
};
myProcess.Start();
myProcess.BeginOutputReadLine();
myProcess.StandardInput.WriteLine("solve;");
myProcess.StandardInput.WriteLine("print 'commandDone';");
eventEnd.WaitOne();

The best option would be to use the Processs.OutputDataReceived event instead of a tight while loop. It’s like the event async pattern, you launch an asynchronous task and wait for an event callback telling you it’s done. The continuation of the asynchronous task would go in the event handler. Remember to unsubscribe the event handler the first time it goes off, otherwise it will be firing when you don’t want it to.
Another option I have never tried is Process.WaitForInputIdle() method, but I’m not sure if this will work in your particular case. If it does you wouldn’t need to write anything to the input stream.

cant get process error output using process.ErrorDataReceived c#

I've built Form App that I use for some time , Now I want to Catch the StandardError of my process as well as its standartOutput
I've looked at answers in SO and MSDN and yet and cant get it right
My code :
public void RunProcess(string FileName, string Arguments,, bool IsPrintOutput = true)
{
process = new Process();
process.ErrorDataReceived += new DataReceivedEventHandler(OnDataReceivedEvent);
if (IsPrintOutput) process.OutputDataReceived += new DataReceivedEventHandler(OnDataReceivedEvent);
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.CreateNoWindow = true;
process.StartInfo.UseShellExecute = false;
process.StartInfo.FileName = FileName;
process.StartInfo.Arguments = Arguments;
if (EventWhenExit)
{
process.EnableRaisingEvents = true;
process.Exited += new EventHandler(myprocess_Exited);
}
process.Start();
process.BeginOutputReadLine();
//run polling on stored logs to print them to screen
PollingService();
}
I've check it with Iperf and I see that when I run it with correct argument I get correct output
but when I just send it with out any argumnet I see that with cmd I get
C:\>iperf.exe
Usage: iperf [-s|-c host] [options]
Try `iperf --help' for more information.
And my App I get Nothing !
what am I missing here ?
Thanks
You can stop reading here ! If you want to see the details of inner method continue below :
private void OnDataReceivedEvent(object sender, DataReceivedEventArgs e)
{
string ProcessOutput = e.Data;
ProcessLog.Add(e.Data);
}
private void PollingService()
{
var T = new Thread (()=>
{
while (true /* ProcessRunning*/)
{
if (ProcessLogIndex < ProcessLog.Count)
{
lock (this)
{
var tempList = ProcessLog.GetRange(ProcessLogIndex, ProcessLog.Count - ProcessLogIndex);
ProcessLogIndex = ProcessLog.Count;
foreach (var ToSend in tempList)
{
onDataOutputFromProcess(this, ToSend, sProcessNameID.ToString());
}
}
}
Thread.Sleep(400);
}
});
T.IsBackground = true;
T.Start();
}

I don't see a call to BeginErrorReadLine() anywhere in the code you posted. If you don't call that method, then the Process class won't actually redirect the stderr to your event handler.
I believe the above is the issue, but if you are actually calling that somewhere (and just didn't show it), then it is worth considering that there are some strange console programs out there that don't actually used stderr (or stdout) for error output. Instead, they write directly to the console window or some other non-standard mechanism. In those cases, you won't be able to receive the error output by redirecting stderr.
You can identify those programs by redirecting their output at the command like with e.g. iperf.exe 2> foo.txt. The stderr file handle is 2, and so that syntax redirects that file handle to a file named foo.txt. If the file is empty and you see errors on the screen, then the program is one of those strange programs.
But really, I think you probably just forgot to call BeginErrorReadLine(). :)

How to get the ExitCode of a running process

I'm writing an application to check the exit code of another application. The application I am monitoring may already be running so I'm checking for it with Process.GetProcessesByName. If it exists I'm checking the exit code after a call to WaitForExit but when I do I get an exception:
"Process was not started by this object, so requested information cannot be determined."
If I start the process (if it isn't already running) then it doesn't give me the exception.
(Windows 8.1)
So how do I find out what the ExitCode was when I haven't started the process? The only option I can think of is to write an output code to a text file on exit and read that in...

System.Diagnostics.Process exposes events that you can access after setting EnableRaisingEvents to true:
int processId = 0; // TODO: populate this variable
var proc = System.Diagnostics.Process.GetProcessById(processId);
proc.EnableRaisingEvents = true;
proc.Exited += ProcessEnded;
Event handler:
private void ProcessEnded(object sender, EventArgs e)
{
var process = sender as Process;
if (process != null)
{
var test = process.ExitCode;
}
}
variable test now contains the exit code.
Tested on Windows 8.1

Restart an application by itself

I want to build my application with the function to restart itself. I found on codeproject
ProcessStartInfo Info=new ProcessStartInfo();
Info.Arguments="/C choice /C Y /N /D Y /T 3 & Del "+
Application.ExecutablePath;
Info.WindowStyle=ProcessWindowStyle.Hidden;
Info.CreateNoWindow=true;
Info.FileName="cmd.exe";
Process.Start(Info);
Application.Exit();
This does not work at all...
And the other problem is, how to start it again like this?
Maybe there are also arguments to start applications.
Edit:
http://www.codeproject.com/script/Articles/ArticleVersion.aspx?aid=31454&av=58703

I use similar code to the code you tried when restarting apps. I send a timed cmd command to restart the app for me like this:
ProcessStartInfo Info = new ProcessStartInfo();
Info.Arguments = "/C ping 127.0.0.1 -n 2 && \"" + Application.ExecutablePath + "\"";
Info.WindowStyle = ProcessWindowStyle.Hidden;
Info.CreateNoWindow = true;
Info.FileName = "cmd.exe";
Process.Start(Info);
Application.Exit();
The command is sent to the OS, the ping pauses the script for 2-3 seconds, by which time the application has exited from Application.Exit(), then the next command after the ping starts it again.
Note: The \" puts quotes around the path, incase it has spaces, which cmd can't process without quotes.
Hope this helps!

Why not use
Application.Restart();
??
More on Restart

Why not just the following?
Process.Start(Application.ExecutablePath);
Application.Exit();
If you want to be sure the app does not run twice either use Environment.Exit(-1) which kills the process instantaneously (not really the nice way) or something like starting a second app, which checks for the process of the main app and starts it again as soon as the process is gone.

You have the initial application A, you want to restart.
So, When you want to kill A, a little application B is started, B kill A, then B start A, and kill B.
To start a process:
Process.Start("A.exe");
To kill a process, is something like this
Process[] procs = Process.GetProcessesByName("B");
foreach (Process proc in procs)
proc.Kill();

A lot of people are suggesting to use Application.Restart. In reality, this function rarely performs as expected. I have never had it shut down the application I am calling it from. I have always had to close the application through other methods such as closing the main form.
You have two ways of handling this. You either have an external program that closes the calling process and starts a new one,
or,
you have the start of your new software kill other instances of same application if an argument is passed as restart.
private void Application_Startup(object sender, StartupEventArgs e)
{
try
{
if (e.Args.Length > 0)
{
foreach (string arg in e.Args)
{
if (arg == "-restart")
{
// WaitForConnection.exe
foreach (Process p in Process.GetProcesses())
{
// In case we get Access Denied
try
{
if (p.MainModule.FileName.ToLower().EndsWith("yourapp.exe"))
{
p.Kill();
p.WaitForExit();
break;
}
}
catch
{ }
}
}
}
}
}
catch
{
}
}

Winforms has the Application.Restart() method, which does just that. If you're using WPF, you can simply add a reference to System.Windows.Forms and call it.

Another way of doing this which feels a little cleaner than these solutions is to run a batch file which includes a specific delay to wait for the current application to terminate. This has the added benefit of preventing the two application instances from being open at the same time.
Example windows batch file ("restart.bat"):
sleep 5
start "" "C:\Dev\MyApplication.exe"
In the application, add this code:
// Launch the restart batch file
Process.Start(#"C:\Dev\restart.bat");
// Close the current application (for WPF case)
Application.Current.MainWindow.Close();
// Close the current application (for WinForms case)
Application.Exit();

My solution:
private static bool _exiting;
private static readonly object SynchObj = new object();
public static void ApplicationRestart(params string[] commandLine)
{
lock (SynchObj)
{
if (Assembly.GetEntryAssembly() == null)
{
throw new NotSupportedException("RestartNotSupported");
}
if (_exiting)
{
return;
}
_exiting = true;
if (Environment.OSVersion.Version.Major < 6)
{
return;
}
bool cancelExit = true;
try
{
List<Form> openForms = Application.OpenForms.OfType<Form>().ToList();
for (int i = openForms.Count - 1; i >= 0; i--)
{
Form f = openForms[i];
if (f.InvokeRequired)
{
f.Invoke(new MethodInvoker(() =>
{
f.FormClosing += (sender, args) => cancelExit = args.Cancel;
f.Close();
}));
}
else
{
f.FormClosing += (sender, args) => cancelExit = args.Cancel;
f.Close();
}
if (cancelExit) break;
}
if (cancelExit) return;
Process.Start(new ProcessStartInfo
{
UseShellExecute = true,
WorkingDirectory = Environment.CurrentDirectory,
FileName = Application.ExecutablePath,
Arguments = commandLine.Length > 0 ? string.Join(" ", commandLine) : string.Empty
});
Application.Exit();
}
finally
{
_exiting = false;
}
}
}

This worked for me:
Process.Start(Process.GetCurrentProcess().MainModule.FileName);
Application.Current.Shutdown();
Some of the other answers have neat things like waiting for a ping to give the initial application time to wind down, but if you just need something simple, this is nice.

For .Net application solution looks like this:
System.Web.HttpRuntime.UnloadAppDomain()
I used this to restart my web application after changing AppSettings in myconfig file.
System.Configuration.Configuration configuration = WebConfigurationManager.OpenWebConfiguration("~");
configuration.AppSettings.Settings["SiteMode"].Value = model.SiteMode.ToString();
configuration.Save();