I have made a python console script, and converted it to a .exe with py2exe (I have used the console = ['test.py'] line in my setup file).
The program parses a file, and during the parse it prints out how much of the file it has parsed. Typical output would be:
Processing (currently at 1%)
Processing (currently at 4%)
etc.
When running the file in a cmd window it works just as expected.
I have also created a very small C# WPF program that just runs the parser:
Process p = new Process();
p.StartInfo = new ProcessStartInfo(#"C:\temp\test.exe");
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.UseShellExecute = false;
p.StartInfo.CreateNoWindow = true;
p.OutputDataReceived += new DataReceivedEventHandler(p_OutputDataReceived);
Task t = new Task(() =>
{
p.Start();
p.BeginOutputReadLine();
p.WaitForExit();
p.Close();
});
t.Start();
The p_OutputDataReceived handler just sends the received output to a textbox. This works and I have tested it on other programs, and there I get the output from the program when I expect.
However when I run my parser (the one created with py2exe) I get all outputs just after the parser has finished. So in the end I get all the correct output, but I get them all at the same time...
(note, I don't get one big output, but rather all the expected outputs but still, all at the same time)
So to be perfectly clear here:
If I run the parser from the command window, I get the outputs one by one
I have tested to run a C# console program instead of the py2exe generated program and that works (I get the outputs one by one)
Python checks if sys.stdout (the program's standard output) is a console. If it is, Python flushes write buffers immediately so the user can see it. Otherwise writes get cached and are outputted all at once:
when the write buffer is full or
at program exit.
The logic behind this is better performance, since when redirecting stdout to a file or to other programs, typically no one cares when the output occurs.
You can fix this by including sys.stdout.flush() in strategic locations in your Python parser script (i.e. directly after printing status line).
Btw: You should be able to observe the same time-delay behavior if you redirect the output of your parser to more, for example:
C:\temp\test.exe | more
Related
This question already has an answer here:
Reading from a process, StreamReader.Peek() not working as expected
(1 answer)
Closed 6 years ago.
I am having trouble setting up a c# application that creates and interacts with a python process1. A simplistic example is given below.
EDIT: Further research on SO unveiled that my question is a possible duplicate. A potentially related known bug in the .NET Framework is duscussed here and here. It seems that back in 2014 the only easy workaround is indeed to ask the child process to write something in both stdOut and stdErr. But I'd like to know if this assumption is correct and wonder if there hasn't been a fix since 2014?
I have to fulfill the following boundary conditions:
I am not able to close the python process after handing over a script or a command, but I have to keep the process alive. Edit: For that reason I can not make use of the Process.WaitForExit() Method
As the std's remain open all the time, I believe I can't check for EndOfStream, as that would require to read to the end of the stream, which does not exist.
Furthermore, my application has to wait for the response of the python process, therefore the asynchronous option using BeginOutputReadLine() with OnOutputDataReceived seems not appropriate to me.
As the commands that will be sent to python are arbitrary user input, pythons result might be either in stdOut or stdErr ("4+7" results in "11" stored in stdOut; "4+a" results in "name 'a' is not defined" in stdErr)
What I do is to:
set up a python process in interactive mode (Argument "-i")
enable redirect of StdIn, Out, and Err
start the process
get StreamReaders and Writers for the Std's
After that, I want to initially check the StdOut and StdErr. I know that python writes the following piece of information to the StdErr
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC
v.1500 32 bit (Intel)] on win32
and I am able to get this line by using errorReader.Peek() and reading character-based from the errorReader 2.
However, the situation with another process might be totally different. Even with Python, I run into the following problem: when I want to initially read from the outputReader, there is nothing contained in it and outputReader.Peek() seems to run into a deadlock. As mentioned above, the same holds for outputReader.EndOfStream or outputReader.ReadToEnd(). So how do I know if the stdOut can be used at all without causing a deadlock?
Code:
// create the python process StartupInfo object
ProcessStartInfo _tempProcessStartInfo = new ProcessStartInfo(#"C:\TMP\Python27\python.exe");
// ProcessStartInfo _tempProcessStartInfo = new ProcessStartInfo(PathToPython + "python.exe");
// python uses "-i" to run in interactive mode
_tempProcessStartInfo.Arguments = "-i";
// Only start the python process, but don't show a (console) window
_tempProcessStartInfo.WindowStyle = ProcessWindowStyle.Minimized;
_tempProcessStartInfo.CreateNoWindow = true;
// Enable the redirection of python process std's
_tempProcessStartInfo.UseShellExecute = false;
_tempProcessStartInfo.RedirectStandardOutput = true;
_tempProcessStartInfo.RedirectStandardInput = true;
_tempProcessStartInfo.RedirectStandardError = true;
// Create the python process object and apply the startupInfos from above
Process _tempProcess = new Process();
_tempProcess.StartInfo = _tempProcessStartInfo;
// Start the process
bool _hasStarted = _tempProcess.Start();
//// ASynch reading seems not appropriate to me:
// _tempProcess.BeginOutputReadLine();
// _tempProcess.BeginErrorReadLine();
// Create StreamReaders and Writers for the Std's
StreamReader outputReader = _tempProcess.StandardOutput;
StreamReader errorReader = _tempProcess.StandardError;
StreamWriter commandWriter = _tempProcess.StandardInput;
// Create StringBuilder that collects results and ErrorMessages
StringBuilder tmp = new StringBuilder("");
// Create temp variable that is used to peek into streams. C# uses -1 to indicate that there is no more byte to read
int currentPeek = -1;
// Get Initial Error Message. In this specific case, this is the python version
tmp.AppendLine("INITIAL ERROR MESSAGE:");
currentPeek = errorReader.Peek();
while (currentPeek >= 0)
{
char text = (char)errorReader.Read();
tmp.Append(text);
currentPeek = errorReader.Peek();
}
// Get initial output Message. In this specific case, this is EMPTY, which seems to cause this problem, as ...
tmp.AppendLine("INITIAL STDOUT MESSAGE:");
//// ... the following command CREATES a well defined output, and afterwards everything works fine (?) but ...
//commandWriter.WriteLine(#"print 'Hello World'");
//// ... without the the above command, neither
//bool isEndOfStream = outputReader.EndOfStream;
//// ... nor
// currentPeek = outputReader.Peek();
//// ... nor
// tmp.AppendLine(outputReader.ReadLine());
//// ... nor
//tmp.AppendLine(outputReader.ReadToEnd());
//// ... works
// Therefore, the following command creates a deadlock
currentPeek = outputReader.Peek();
while (currentPeek >= 0)
{
char text = (char)outputReader.Read();
tmp.Append(text);
currentPeek = errorReader.Peek();
}
_currentPythonProcess = _tempProcess;
return true;
1 An easy fix to this very specific problem is to send a valid command to the process first, for example simply "4", which returns a "4" as well... However, I want to understand how process streams, pipes and the corresponing readers and writers work and how I can use them in C#. Who knows what future brings, maybe I run into buffer problems when pythons response is 2^n+1 bytes long...
2 I know that I can also read line-based. However, the Peek() prevents me from reporting problems that are related to truncated lines.
If you can wait for the process to end and then read the buffers, you might be able to use Process.WaitForExit. There is also another method you can check, Process.WaitForInputIdle, but it depends on the process having a message loop, which I don't think a Python script gets when executing.
I'm trying to write a C# program that captures the standard output in a python program. My problem is that all of the output comes after the program has executed rather than when it actually happens. As an example, for this python program:
print "Hello"
time.sleep(2)
print "Hello"
I would expect to get "Hello", a two second gap, and then another "Hello". The actual result is a two second gap and then "Hello", "Hello".
If I run the above python script from the command line, I get the desired behaviour. If the command prompt can do this, then I should be able to mimic that functionality without having to flush the buffer repeatedly.
I'm using this to run the process from C#:
_proc = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = "C:\\Python27\\python.exe",
Arguments = pyScript,
RedirectStandardError = true,
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
}
};
_proc.OutputDataReceived += ProcOnOutputDataReceived;
_proc.Start();
_proc.BeginOutputReadLine();
I can run this C# code (and changing the ProcessStartInfo properties above to run C# executable) and it behaves correctly:
Console.WriteLine("Hello");
Thread.Sleep(2000);
Console.WriteLine("Hello");
With this code I get "Hello", a two second gap, and then another "Hello".
Any idea why? How can I get the python interpreter to send the standard output as it happens?
I know this is old but running python with -u option (unbuffered) seems to be what you were after.
Unbuffered will not wait before flushing output.
Running python without this option, output is flushed when:
The buffer is full
You call sys.stdout.flush()
The process ends
You need to flush the output buffer.
import sys
sys.stdout.flush()
See this question: How to flush output of Python print?
Good
(Capture standard output of a python program as it happens)
Em console is NEED:
import sys
sys.stdout.flush()
Our C# (V3.5) application needs to call another C++ executable which is from another company. we need to pass a raw data file name to it, it will process that raw data (about 7MB) file and generate 16 result files (about 124K for each).
The code to call that executable is this:
ProcessStartInfo startInfo = new ProcessStartInfo();
startInfo.CreateNoWindow = true;
startInfo.UseShellExecute = false;
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardOutput = true;
startInfo.WindowStyle = ProcessWindowStyle.Hidden;
startInfo.FileName = exePath;
startInfo.Arguments = rawDataFileName;
try
{
Process correctionProcess = Process.Start(startInfo);
correctionProcess.WaitForExit();
}
catch(nvalidOperationException ex)
{
....
}
catch(...)
...
It works fine. Now we have new raw data. After replace the old raw data with the new raw data file. That executable process never return to us. It will hang forever. If we kill our C# application, those result files will be generated in the target directoy. It looks like the executable does create those result files but has issue to write to the disk and return to us until the process is terminated.
It is NOT like this with the old raw data file.
When we run the executable with the new raw data directly (no from our C# app call), it works fine. This means this executable has no problem with the new raw data.
My question 1: what's the possible causes for this behaviour?
Now I change our code with startInfo.UseShellExecute = true; and add startInfo.WorkingDirectory= ..., and disabled
//startInfo.RedirectStandardError = true;
//startInfo.RedirectStandardOutput = true;
Then it works.
My question 2: why use Windows Shell solve this issue?
My question 3: why it works before without using Shell?
My question 4: when we should use Shell and When shouldn't?
thanks,
Several possibilities:
You are redirecting output and error but not reading it. The process will stall when its stdout or stderr buffer fills up to capacity.
The program might be displaying an error message and waiting for a keypress. You are not redirecting input nor check stderr, that keypress will never come.
Some programs, xcopy.exe is a very good example, require stdin to be redirected when you redirect stdout. Although the failure mode for xcopy.exe is an immediate exit without any diagnostic.
Seeing it fixed when you kill your C# program makes the first bullet the likeliest reason.
I know this, it is a very common problem. I has to do with the output, which must be handled asynchronously. You just can't WaitForExit when output exceeds certain amount of data.
You need to add
myStdErr= correctionProcess.StandardError.ReadToEnd();
Only once usually works, if you want to overkill this works ("P" being my Process)
while (!P.HasExited)
stdErr+= P.StandardError.ReadToEnd();
If you don't need the stdout/stderr, just turn the Redirect* properties to false.
I'm currently trying to get the output of an executable console-app into an other one. To be exact, a little overview of what I'm trying to do:
I have one executable which I cannot edit and neither see it's code. It writes some (quite a bunch to be honest) lines into the console when executed.
Now I want to write another executable that starts the one above and reads the things it writes.
Seems simple to me, so I started coding but ended up with an error message saying that StandardOut has not been redirected or the process hasn't started yet.
I tried it using this kinda structure (C#):
Process MyApp = Process.Start(#"C:\some\dirs\foo.exe", "someargs");
MyApp.Start();
StreamReader _Out = MyApp.StandardOutput;
string _Line = "";
while ((_Line = _Out.ReadLine()) != null)
Console.WriteLine("Read: " + _Line);
MyApp.Close();
I can open the executable and it also does open the one inside, but as soon as it comes to reading the returned values, the app crashes.
What am I doing wrong?!
Take a look at the documentation for the Process.StandardOutput property. You will need to set a boolean indicating that you want the stream redirected as well as disabling shell execute.
Note from the documentation:
To use StandardOutput, you must set ProcessStartInfo..::.UseShellExecute to false, and you must set ProcessStartInfo..::.RedirectStandardOutput to true. Otherwise, reading from the StandardOutput stream throws an exception
You would need to change your code a little bit to adjust for the changes:
Process myApp = new Process(#"C:\some\dirs\foo.exe", "someargs");
myApp.StartInfo.UseShellExecute = false;
myApp.StartInfo.RedirectStandardOutput = false;
myApp.Start();
string output = myApp.StandardOutput.ReadToEnd();
p.WaitForExit();
you could try setting processStartInfo.RedirectStandardOutput = true;
As noted above, you can use RedirectStandardOutput as here.
Another, dirtier way is something like
using (Process child = Process.Start
("cmd", #"/c C:\some\dirs\foo.exe someargs > somefilename"))
{
exeProcess.WaitForExit();
}
And then read its output from somefilename
I am trying to run a process in c# using the Process class.
Process p1 = new process();
p1.startinfo.filename = "xyz.exe";
p1.startinfo.arguments = //i am building it based on user's input.
p1.start();
So based on user input i am building the argument value. Now i have a case where i have to pipe the output of p1 to another process say grep. so i basically tried this
p1.startinfo.arguments = "-info |grep 1234" ;
what i intended is something like xyz.exe -info|grep 1234
but this doesn't seem to work in .net .. I can actually create another process variable and run "grep" as a separate process.. But i was wondering if there is any way to do as iam trying out above..
The much easier way would be to do just use cmd as your process.
Process test = new Process();
test.StartInfo.FileName = "cmd";
test.StartInfo.Arguments = #"/C ""echo testing | grep test""";
test.Start();
You can capture the output or whatever else you want like any normal process then. This was just a quick test I built, but it works outputting testing to the console so I would expect this would work for anything else you plan on doing with the piping. If you want the command to stay open then use /K instead of /C and the window will not close once the process finishes.