We Keep Coding

Buffering in redirected STDOUT for a child process

Code like this can host a console app and listen to its output to STDOUT AND STDERR
Process process = new Process();
process.StartInfo.FileName = exePath;
process.StartInfo.UseShellExecute = false;
process.StartInfo.WorkingDirectory = context.WorkingDirectory;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.RedirectStandardInput = true; // if you don't and it reads, no more events
process.StartInfo.UseShellExecute = false;
process.StartInfo.CreateNoWindow = false;
process.EnableRaisingEvents = true;
process.ErrorDataReceived += (sender, dataReceivedEventArgs) =>
{
lastbeat = DateTime.UtcNow;
if (dataReceivedEventArgs.Data != null)
{
if (dataReceivedEventArgs.Data.EndsWith("%"))
{
context.Logger.Information($" PROGRESS: {dataReceivedEventArgs.Data}");
}
else
{
msg.Append(" STDERR (UNHANDLED EXCEPTION): ");
msg.AppendLine(dataReceivedEventArgs.Data);
success = false;
}
}
};
process.OutputDataReceived += (sender, dataReceivedEventArgs) =>
{
lastbeat = DateTime.UtcNow;
if (dataReceivedEventArgs.Data != null)
{
if (dataReceivedEventArgs.Data.EndsWith("%"))
{
context.Logger.Information($" PROGRESS: {dataReceivedEventArgs.Data}");
}
else
{
context.Logger.Information($" STDOUT: {dataReceivedEventArgs.Data}");
}
}
};
lastbeat = DateTime.UtcNow;
process.Start();
process.BeginErrorReadLine();
process.BeginOutputReadLine();
// wait for the child process, kill it if hearbeats are too slow
while (!process.HasExited)
{
Thread.Sleep(100);
var elapsed = DateTime.UtcNow - lastbeat;
if (elapsed.TotalSeconds > heartbeatIntervalSeconds * 3)
{
success = false;
msg.AppendLine("MODULE HEARTBEAT STOPPED, TERMINATING.");
try
{
process.Kill(entireProcessTree: true); // ...and your children's children
}
catch (Exception ek)
{
msg.AppendLine(ek.Message);
}
}
}
if (success)
{
process.Dispose();
context.Logger.Debug("MODULE COMPLETED");
return JobStepResult.Success;
}
else
{
process.Dispose();
context.Logger.Debug("MODULE ABORTED");
throw new Exception(msg.ToString());
}
The hosted processes are potentially very long running, so we invented a heartbeat mechanism. There's a convention here that STDERR is used for out of band communication so that STDOUT isn't polluted with heartbeat messages. Any line of text written to STDERR that ends with a percent sign is treated as a heartbeat, everything else is a normal error message.
We have two hosted modules, one of which works perfectly with heartbeats received in a timely manner, but the other seems to hang until all its output on both STDERR and STDOUT arrives in a flood.
The hosted modules are written in Lahey FORTRAN. I have no visibility on that code. I have suggested to the author that she may need to flush her output streams or possibly yield using whatever is the FORTRAN equivalent to Thread.Sleep(10);
However, it is not beyond the realms of possibility that the problem is on my side. When the modules are executed manually in a console their output appears at a steady pace with heartbeat messages appearing in a timely manner.
What governs the behaviour of captured streams?
Are they buffered?
Is there any way to influence this?
This may be related. Get Live output from Process
It seems (see comments) that this is an old problem. I pulled my hosting code out into a console app and the problem is manifest there too.
Codiçil
This doesn't happen when the hosted console app is a dotnet core app. Presumably dotnet core apps use ConPTY because that way they can work the same across platforms.

This is an old problem. An application can detect if they are running in a console and if not, choose to buffer their output. For example, this is something that the microsoft C runtime does deliberately whenever you call printf.
An application should not buffer writes to stderr, as errors should be made available before your program has a chance to crash and wipe any buffer. However, there's nothing to force that rule.
There are old solutions to this problem. By creating an off-screen console you can detect when output is written to the console buffer. There's a write up on Code Project that talks about this issue and solution in more detail.
More recently, Microsoft has modernised their console infrastructure. If you write to a modern console, your output is converted to a UTF-8 stream with embedded VT escape sequences. A standard that the rest of the world has been using for decades. For more details you can read their blog series on that work here.
I believe that it should be possible to build a new modern workaround. A stub process, similar to the Code Project link above, that uses these new pseudo console API's to launch a child process, capture console I/O and pipe that output unbuffered to its own stdio handles. I would be nice if such a stub process were distributed in windows, or by some other 3rd party, but I haven't found one yet.
There are a couple of samples available in github.com/microsoft/terminal that would be a good starting point if you wanted to create your own stub.
However I believe that using either this solution, or the old workaround would still merge the output and error streams together.

Process ends prematurely when executed from a GUI, doesn't when executed via a Console testing app

I'm sorry if the title is not matching the precise description of the issue I'm facing, I accept suggestions for edits.
I'm developing a component that, when called, shall start a Process and wait for it to complete. The Process might want to redirect std output to a custom winform (a popup) for the user to have a feedback on the Process status.
In order to test the component before implementing it in the official application, I've developed a Console app that calls the module, in the same way the official application will do accepting user input from the GUI --> not the popup, the application main window.
When tested in the Console app, everything works as I expect. When testing in the official GUI application, the Process ends prematurely, with exit code 0 (all fine). I mean, it prints something in the user winform, up to a certain point, always the same point, then exits.
I can't understand why or how to identify root causes and maybe do something about it. I'm looking for help, hereafter the module source code for process execution:
ProcessStartInfo processStartInfo = new ProcessStartInfo();
processStartInfo.FileName = myExePath;
processStartInfo.Arguments = myExeArguments;
processStartInfo.RedirectStandardOutput = true;
processStartInfo.RedirectStandardError = false;
processStartInfo.UseShellExecute = false;
processStartInfo.CreateNoWindow = true;
using (Process process = Process.Start(processStartInfo))
{
process.EnableRaisingEvents = true;
// accessory code. Issue is reproduced even without this section ---- //
Thread t = new Thread(() =>
{
// show a custom winform for the user to read
});
t.Start();
// ------------------------------------------ //
process.OutputDataReceived += (s, a) => {
// Do something with a.Data
};
process.BeginOutputReadLine();
process.WaitForExit();
}
EDIT:
let's say that the Process myExe should print something to stdout. I read this something and it goes along these lines:
line 1
line 2
I'm doing something
line 3
start doing something else <-- when component is integrated in official application, here Process Exits without exceptions
line 4
line 5 <-- When component is tested in Console app, here is where Process Exits without exceptions.
I should read up to line 5 in both testing scenario and official scenario.

C# process see output in console and get a copy of it at the same time

I'm writing an application that does analysis on other applications,
the user can choose which application he wants to analyze and run it...
a console would be opened and he'll see his application progress (output) and will be able to stop his application at anytime (by closing the console).
on my side (the application that runs the analysis), I need to get the user's application output (stdout & stderr) and run some parsing on it.
so to sum up:
(1) users' application must run in a new console window so that the user could stop the analysis at any point by closing the new console.
(2) user must be able to see his application output in the console to know when to stop it.
(3) the "parent" application will need a copy of that output (doesn't matter real time or not).
tried redirecting streams - makes (3) work but (2) not..
so how can I achieve all 3 requests above at the same time in a single run?
any ideas??

ok figured it out, if anyone is interested in the solution:
StringBuilder outStr = new StringBuilder("");
StringBuilder errStr = new StringBuilder("");
process.Start();
process.OutputDataReceived += (sendingProcess, outLine) =>
{
outStr.AppendLine(outLine.Data);
Console.Out.WriteLine(outLine.Data); // echo the output
};
process.ErrorDataReceived += (sendingProcess, errorLine) =>
{
errStr.AppendLine(errorLine.Data);
Console.Error.WriteLine(errorLine.Data); // echo the error
};
process.BeginOutputReadLine();
process.BeginErrorReadLine();

C# Run a program with progressbar to compress listbox items

I would like to write a custom gui for a compressor.
How is it possible to run the Command Line Interface on all files added into listbox ?
Also how to display a progressbar displaying the current progress (not in % but at least which file is currently processing).

If you're talking about just running a command line program, you can use System.Diagnostic.Process.
System.Diagnostics.Process p = new System.Diagnostics.Process();
p.StartInfo.FileName = #"c:\path\to\my.exe";
p.StartInfo.Arguments = #"c:\path\to\filetoCompress.file";
//event wireup
p.Exited += new EventHandler(p_exited);
p.Start();
As far as displaying progress for that, you could use the Exited event of the process to wire up to a process bar that shows the progress of number exted vs. number total.
void p_Exited(object sender, EventArgs e)
{
//assuming you have setup this bar previously.
pbFilesCommpressProgress.PerformStep();
}

In a CLI you can use one of many ASCII characters like ||| or ... and then you can append them as files progress.. or you can denote (file x of xxx), and inform the user of the progress as it occurs.

Console application prompting for input

I'm trying to put together a wrapper around a console application using StandardInput and StandardOutput. I'm getting stuck where the console application would prompt for input such as a password.
I'd like to read from StandardOutput, prompt the user using the read text, and write the user's input back to the console application using its StandardInput. Seems easy enough, here's what I have currently:
Process process = new Process()
{
StartInfo =
{
FileName = "bin\\vpnc.exe",
Arguments = "--no-detach --debug 0",
CreateNoWindow = true,
UseShellExecute = false,
RedirectStandardInput = true,
RedirectStandardOutput = true,
}
};
process.OutputDataReceived += (s, args) =>
{
textBlock1.Dispatcher.Invoke(new Action(() =>
{
textBlock1.Text += args.Data;
}));
};
process.Start();
process.BeginOutputReadLine();
The problem is that BeginOutputReadLine() is doing just that...waiting for a line ending. In this case it just sits, and sits, and sits because there is no line to read...the console application has written out text with no line ending and is waiting for input. Coincidentally, when I manually kill the process the event fires and I get the text.
Is there a way to tell that the process is waiting for StandardInput? Or am I missing a completely obvious way to accomplish the goal?

Unless you need something asynchronous you probably want ReadToEnd.
Here is a list of all StreamReader Methods

process.StandardOutput.BaseStream.BeginRead(...) is a potential substitute for your readline, and that will not wait for a line ending however you'd need to know what terminates the output to figure out when not to start wait for the next chunk of data

As Rune said, you can access directly to the output stream of the process (process.StandardOutput) and read from there (if you don't want to wait until a line break is entered into the console app), but this means that you need to check periodically for new data.
To interact with the application, you can just write to the StandardInput of the process (create a StreamWriter that writes to the process.StandardInput).
A nice sample of writing to it is on the MSDN documentation (http://msdn.microsoft.com/en-us/library/system.diagnostics.process.beginoutputreadline.aspx).
Hope this helps

You need to use the synchronous read method and handle any necessary threading yourself. The below code won't tell you that input is expected, but you will be able to detect that a prompt is displayed.
char[] b = new char[1024];
while (!process.HasExited) {
int c = process.StandardOutput.Read(b, 0, b.Length);
context.Response.Write(b, 0, c);
Thread.Sleep(100);
}