We Keep Coding

Buffering in redirected STDOUT for a child process

Code like this can host a console app and listen to its output to STDOUT AND STDERR
Process process = new Process();
process.StartInfo.FileName = exePath;
process.StartInfo.UseShellExecute = false;
process.StartInfo.WorkingDirectory = context.WorkingDirectory;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.RedirectStandardInput = true; // if you don't and it reads, no more events
process.StartInfo.UseShellExecute = false;
process.StartInfo.CreateNoWindow = false;
process.EnableRaisingEvents = true;
process.ErrorDataReceived += (sender, dataReceivedEventArgs) =>
{
lastbeat = DateTime.UtcNow;
if (dataReceivedEventArgs.Data != null)
{
if (dataReceivedEventArgs.Data.EndsWith("%"))
{
context.Logger.Information($" PROGRESS: {dataReceivedEventArgs.Data}");
}
else
{
msg.Append(" STDERR (UNHANDLED EXCEPTION): ");
msg.AppendLine(dataReceivedEventArgs.Data);
success = false;
}
}
};
process.OutputDataReceived += (sender, dataReceivedEventArgs) =>
{
lastbeat = DateTime.UtcNow;
if (dataReceivedEventArgs.Data != null)
{
if (dataReceivedEventArgs.Data.EndsWith("%"))
{
context.Logger.Information($" PROGRESS: {dataReceivedEventArgs.Data}");
}
else
{
context.Logger.Information($" STDOUT: {dataReceivedEventArgs.Data}");
}
}
};
lastbeat = DateTime.UtcNow;
process.Start();
process.BeginErrorReadLine();
process.BeginOutputReadLine();
// wait for the child process, kill it if hearbeats are too slow
while (!process.HasExited)
{
Thread.Sleep(100);
var elapsed = DateTime.UtcNow - lastbeat;
if (elapsed.TotalSeconds > heartbeatIntervalSeconds * 3)
{
success = false;
msg.AppendLine("MODULE HEARTBEAT STOPPED, TERMINATING.");
try
{
process.Kill(entireProcessTree: true); // ...and your children's children
}
catch (Exception ek)
{
msg.AppendLine(ek.Message);
}
}
}
if (success)
{
process.Dispose();
context.Logger.Debug("MODULE COMPLETED");
return JobStepResult.Success;
}
else
{
process.Dispose();
context.Logger.Debug("MODULE ABORTED");
throw new Exception(msg.ToString());
}
The hosted processes are potentially very long running, so we invented a heartbeat mechanism. There's a convention here that STDERR is used for out of band communication so that STDOUT isn't polluted with heartbeat messages. Any line of text written to STDERR that ends with a percent sign is treated as a heartbeat, everything else is a normal error message.
We have two hosted modules, one of which works perfectly with heartbeats received in a timely manner, but the other seems to hang until all its output on both STDERR and STDOUT arrives in a flood.
The hosted modules are written in Lahey FORTRAN. I have no visibility on that code. I have suggested to the author that she may need to flush her output streams or possibly yield using whatever is the FORTRAN equivalent to Thread.Sleep(10);
However, it is not beyond the realms of possibility that the problem is on my side. When the modules are executed manually in a console their output appears at a steady pace with heartbeat messages appearing in a timely manner.
What governs the behaviour of captured streams?
Are they buffered?
Is there any way to influence this?
This may be related. Get Live output from Process
It seems (see comments) that this is an old problem. I pulled my hosting code out into a console app and the problem is manifest there too.
Codiçil
This doesn't happen when the hosted console app is a dotnet core app. Presumably dotnet core apps use ConPTY because that way they can work the same across platforms.

This is an old problem. An application can detect if they are running in a console and if not, choose to buffer their output. For example, this is something that the microsoft C runtime does deliberately whenever you call printf.
An application should not buffer writes to stderr, as errors should be made available before your program has a chance to crash and wipe any buffer. However, there's nothing to force that rule.
There are old solutions to this problem. By creating an off-screen console you can detect when output is written to the console buffer. There's a write up on Code Project that talks about this issue and solution in more detail.
More recently, Microsoft has modernised their console infrastructure. If you write to a modern console, your output is converted to a UTF-8 stream with embedded VT escape sequences. A standard that the rest of the world has been using for decades. For more details you can read their blog series on that work here.
I believe that it should be possible to build a new modern workaround. A stub process, similar to the Code Project link above, that uses these new pseudo console API's to launch a child process, capture console I/O and pipe that output unbuffered to its own stdio handles. I would be nice if such a stub process were distributed in windows, or by some other 3rd party, but I haven't found one yet.
There are a couple of samples available in github.com/microsoft/terminal that would be a good starting point if you wanted to create your own stub.
However I believe that using either this solution, or the old workaround would still merge the output and error streams together.

Process ends prematurely when executed from a GUI, doesn't when executed via a Console testing app

I'm sorry if the title is not matching the precise description of the issue I'm facing, I accept suggestions for edits.
I'm developing a component that, when called, shall start a Process and wait for it to complete. The Process might want to redirect std output to a custom winform (a popup) for the user to have a feedback on the Process status.
In order to test the component before implementing it in the official application, I've developed a Console app that calls the module, in the same way the official application will do accepting user input from the GUI --> not the popup, the application main window.
When tested in the Console app, everything works as I expect. When testing in the official GUI application, the Process ends prematurely, with exit code 0 (all fine). I mean, it prints something in the user winform, up to a certain point, always the same point, then exits.
I can't understand why or how to identify root causes and maybe do something about it. I'm looking for help, hereafter the module source code for process execution:
ProcessStartInfo processStartInfo = new ProcessStartInfo();
processStartInfo.FileName = myExePath;
processStartInfo.Arguments = myExeArguments;
processStartInfo.RedirectStandardOutput = true;
processStartInfo.RedirectStandardError = false;
processStartInfo.UseShellExecute = false;
processStartInfo.CreateNoWindow = true;
using (Process process = Process.Start(processStartInfo))
{
process.EnableRaisingEvents = true;
// accessory code. Issue is reproduced even without this section ---- //
Thread t = new Thread(() =>
{
// show a custom winform for the user to read
});
t.Start();
// ------------------------------------------ //
process.OutputDataReceived += (s, a) => {
// Do something with a.Data
};
process.BeginOutputReadLine();
process.WaitForExit();
}
EDIT:
let's say that the Process myExe should print something to stdout. I read this something and it goes along these lines:
line 1
line 2
I'm doing something
line 3
start doing something else <-- when component is integrated in official application, here Process Exits without exceptions
line 4
line 5 <-- When component is tested in Console app, here is where Process Exits without exceptions.
I should read up to line 5 in both testing scenario and official scenario.

Strange standard output behavior when blocking thread

I have a C# application that creates a thread which talks to Node.js (node.exe) over standard input.
The code to create this is pretty standard:
ProcessStartInfo NodeStart = new ProcessStartInfo();
NodeStart.FileName = FileName; // node.exe
NodeStart.Arguments = Arguments;
NodeStart.CreateNoWindow = true;
NodeStart.RedirectStandardError = true;
NodeStart.RedirectStandardInput = true;
NodeStart.RedirectStandardOutput = true;
NodeStart.UseShellExecute = false;
process = new Process();
process.OutputDataReceived += ReceivedOutput;
process.StartInfo = NodeStart;
process.Start();
process.BeginOutputReadLine();
The code works completely OK as long as I do not block the thread that is running it by using Thread.WaitOne(). As soon as I call Thread.WaitOne() something goes wrong with the input/output of the process. Next time I unblock the thread and write to standard input, I get nothing back. I inspected the process object in the debugger prior to when the standard input gets written and nothing appears to be wrong (node.exe is running and accepts the standard input).
It may simply be the case that standard input/output breaks when the thread blocks, but is this true? I couldn't find the answer anywhere.

Determine if process is waiting for user input [duplicate]

I am using the Process class to run an exe.
The exe is a 3rd party console application that I do not control.
I wish to know whether the process is waiting for input on the command line.
Should it make any difference, I intend to kill the application should it be waiting for input.
There are suitable events for when there is output from the program waiting to be read, but I cannot see anything similar for when the process is waiting patiently for input.
ProcessStartInfo info = new ProcessStartInfo();
info.FileName = "myapp.exe";
info.CreateNoWindow = true;
info.UseShellExecute = false;
info.RedirectStandardError = true;
info.RedirectStandardInput = true;
info.RedirectStandardOutput = true;
process.StartInfo = info;
process.OutputDataReceived += new DataReceivedEventHandler(process_OutputDataReceived);
process.ErrorDataReceived += new DataReceivedEventHandler(process_ErrorDataReceived);
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
process.WaitForExit();
How do I detect that my process is waiting for input?

Depending on what the 3rd party process is doing exactly you could try polling its threads' states:
foreach(ProcessThread thread in process.Threads)
if (thread.ThreadState == ThreadState.Wait
&& thread.WaitReason == ThreadWaitReason.UserRequest)
process.Kill();
Failing that... you can try to
process.StandardInput.Close();
after calling Start(), I conjecture that an exception will be raised in the child process if it's trying to read from standard input.

If the console application has some sort of prompt waiting for input, you could periodically parse out the console output text using the Process.StandardOutput property of the process and wait for said prompt. Once the proper string is detected, you know that it's waiting for input. See http://msdn.microsoft.com/en-us/library/system.diagnostics.process.standardoutput.aspx.

Use process.StandardInput.writrLine("input");
for sending input to consol in c#

Console information to window

I already know how to catch standard output of a console window, BUT my problem is the case when I get the process with GetProcesses/orByName and do not Start() it myself. Here is the code:
public ProcessCaller(ISynchronizeInvoke isi, Process MárFutóAlkalmazás)
: this(isi)
{
//alapbeállítások
FileName = MárFutóAlkalmazás.StartInfo.FileName;
Arguments = MárFutóAlkalmazás.StartInfo.Arguments;
WorkingDirectory = MárFutóAlkalmazás.StartInfo.WorkingDirectory;
//egyedi beállítások
process = MárFutóAlkalmazás;
process.EnableRaisingEvents = true;
process.StartInfo.CreateNoWindow = true;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
new MethodInvoker(ReadStdOut).BeginInvoke(null, null);
new MethodInvoker(ReadStdErr).BeginInvoke(null, null);
//események
StdErrReceived += new DataReceivedHandler(Loggolás);
StdOutReceived += new DataReceivedHandler(Loggolás);
//kilépés jelzése
process.Exited += new EventHandler(OnKilépés);
}
So this method gets and already running application as MárFutóAlkalmazás parameter. Sets some internal properties, then hooks to Output. However when it comes to
StdOutReceived += new DataReceivedHandler(Loggolás);
and the program runs the Loggolás method to take the console data, it says that the StandardOut is not set, or the process is not started.
Well:
StandardOut is set
Process is running, since I get it by GetProcesses
In this routine I do NOT use process.Start() - since it is started already
Looking for help. Thank yas:
Péter

Ok, so after asking around and checking on net, I learned that you can not hook on an output not started by you. So if your executor application crashes, you will have to restart console application to be able to capture output. You need the .Start().
Actually I see only one salvation for this problem: starting with the ">filename.txt" or such output redirecting parameter. This will stuff everything into a file, so even if executor application crashes you can "reconnect" if you "read only". Did not tested this yet, but I see no other way.