I am redirecting Process.StandardOutput and Process.StandardError from a System.Diagnostics.Process that uses 7zip to extract and zip archives and am unable to read the progress from the process.
It appears, 7Zip like some other applications, emit are backspace and delete characters to partially write a line data and then and delete the written characters using backspace and delete in order o show progress. I am trying to read those partial line outputs from the target process am unable to do so. However, I may be wrong in this assumption.
What I have tried:
var process = new Process
{
StartInfo =
{
FileName = command,
Arguments = arguments,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
process.Start();
After the above code block, I have tried using various methods of reading the data.
I have tried the async event handlers:
process.OutputDataReceived += (sender, args) => { Console.WriteLine(args.Data); };
process.ErrorDataReceived += (sender, args) => { Console.WriteLine(args.Data); };
process.BeginOutputReadLine();
process.BeginErrorReadLine();
I have tried using the async methods of the StandardOutput:
while (!process.StandardOutput.EndOfStream)
{
char[] buffer = new char[256];
int read = process.StandardOutput.ReadAsync(buffer, 0, buffer.Length).Result;
Console.Write(buffer, 0, read);
}
and
process.StandardOutput.BaseStream.CopyToAsync(Console.OpenStandardOutput());
And have tried using the async methods of the underlying base stream.
while (!process.StandardOutput.EndOfStream)
{
byte[] buffer = new byte[256];
int read = process.StandardOutput.BaseStream.ReadAsync(buffer, 0, buffer.Length).Result;
string data = Encoding.UTF8.GetString(buffer, 0, read);
Console.Write(data);
}
As an example, run 7Zip from the terminal with the following command:
"c:\program files\7-zip\7z.exe" x -o"C:\target" "K:\Disk_23339.secure.7z"
This shows progress output when running directly in a command prompt, with each success progress incrementing overwriting the previous:
It then uses backspace chars to overwrite the previous progress.
Running the same command and arguments using Process.Start()
var process = new Process
{
StartInfo =
{
FileName = "c:\program files\7-zip\7z.exe",
Arguments = 'x -o"C:\target" \"K:\Disk_23339.secure.7z\"",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
process.Start();
process.StandardOutput.BaseStream.CopyToAsync(Console.OpenStandardOutput());
process.WaitForExit();
When running this and attempting to read the redirected standard output of the process characters that are not emitted from the source process that do not contain a new line (either by linefeed or carriage return + line feed) are output to the standard output or standard error of System.Diagnostics.Process and hence never written to the console.
7zip of course is just one example. This issue also occurs with numerous PowerShell and Python scripts.
Is there anyway to read these characters from Process.StandardOutput and Process.StandardError.
I am not sure but I think the issue is the underlying stream reader reads one line at a time and never returns these partial lines because they never include line ending characters.
Though the post had been months, in case your problem (not able to get progress from 7zip when execute in command line) still exists, use the "-bsp1" switch in command line argument.
I was also looking for the solution of same issue and just got it (tested successfully in my case) right. By redirect StandardOutput, repeatedly do cmd.StandardOutput.ReadLine() (the Synchronous method, I tested this method instead of asynchronous method that use EventHandler, but I think async method would work too), and use RegExp to detect the progress to update my UI.
My command (run in .NET)
C:\Program Files\7-Zip\7z.exe a -t7z -mx=9 -bsp1 "G:\Temp\My7zTest_Compressed.7z" "G:\SourceFolder"
Credit to #justintjacob
How to show extraction progress of 7zip inside cmd?
Related
My C# program needs to send data to a 3rd-party program via its standard input. However, the program waits for the input stream to reach EOF before processing. Here my code:
// Starts the process.
var process = new Process();
process.StartInfo.CreateNoWindow = true;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.FileName = "foo.exe";
process.Start();
// Sends data to child process.
var input = process.StandardInput;
input.WriteLine("eval('2 * PI')");
input.Flush();
input.Close();
// Reads the result.
var output = process.StandardOutput;
var result = output.ReadLine();
The child program won't do anything and my C# code becomes stuck at output.ReadLine() call. However if I kill the C# process, then the child starts to work exactly on the data I've sent. How can I make the child encounter an EOF while I'm still alive?
StreamWriter might not be sending an actual eof when it closes the stream. You could try writing your own to the stream just before you close it. Something like this might work:
input.Write((char)26);
You may have to find out what the process expects for eof.
I want to be able to read a screen shot of a web site, and am attempting to use phantomjs and ASP.NET.
I have tried using page.render which would save the screen shot to a file. It works as a console application, but not when I call it from an asp.net handler. It is probably due to file permissions, since simple applications (like hello.js) work fine.
That is okay, my preference would be not to write to a file, but to deal with the bytes and return an image directly from the handler.
I am a bit lost as to how to do that. I noticed a method called page.renderBase64, but do not know how to use it.
Currently I am using an IHttpHandler.
There is a similar question here, but that person eventualy dropped phantomjs. I like the look of it and want to continue using it if possible.
Running Phantomjs using C# to grab snapshot of webpage
According to your last comment you can do the following in phantom js file:
var base64image = page.renderBase64('PNG');
system.stdout.write(base64image);
in C#:
var startInfo = new ProcessStartInfo {
//some other parameters here
...
FileName = pathToExe,
Arguments = String.Format("{0}",someParameters),
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true,
WorkingDirectory = pdfToolPath
};
var p = new Process();
p.StartInfo = startInfo;
p.Start();
p.WaitForExit(timeToExit);
//Read the Error:
string error = p.StandardError.ReadToEnd();
//Read the Output:
string output = p.StandardOutput.ReadToEnd();
In your output variable you can read base64 returned from phantomJS and then do what you have planned with it.
Use the wrapper for Phantomjs from here nreco wrapper
You can get js for rastor here : rastorize
And then the following code in C# would do the job.
var phantomJS=new PhantomJS();
phantomJS.Run("rasterize.js", new[] { "http://google.com","ss.pdf" });
This question stemmed from my lack of understanding of what a base64 string actually was.
In the javascript file that phantomjs runs, I can write the base64 image directly to the console like so:
var base64image = page.renderBase64('PNG');
console.log(base64image);
In the c# code that runs phantomjs, I can convert the console output back to bytes and write the image to the response, like so:
var info = new ProcessStartInfo(path, string.Join(" ", args));
info.RedirectStandardInput = true;
info.RedirectStandardOutput = true;
info.UseShellExecute = false;
info.CreateNoWindow = true;
var p = Process.Start(info);
p.Start();
var base64image = p.StandardOutput.ReadToEnd();
var bytes = Convert.FromBase64CharArray(base64image.ToCharArray(), 0, base64image.Length);
p.WaitForExit();
context.Response.OutputStream.Write(bytes, 0, bytes.Length);
context.Response.ContentType = "image/PNG";
This seems to avoid file locking issues I was having.
Using CasperJS coupled with PhantomJS , I've been getting beautiful shots of webpages.
var casper = require('casper').create();
casper.start('http://target.aspx', function() {
this.capture('snapshot.png');
});
casper.run(function() {
this.echo('finished');
});
I highly recommend you check out that tool. I'm still not sure how to do the post-backs though..
Set the 'WorkingDirectory' property of ProcessStartInfo object in order to specify the saving location of the file.
I have tried calling a Process(console application) using the following code:
ProcessStartInfo pi = new ProcessStartInfo();
pi.UseShellExecute = false;
pi.RedirectStandardOutput = true;
pi.CreateNoWindow = true;
pi.FileName = #"C:\fakepath\go.exe";
pi.Arguments = "FOO BAA";
Process p = Process.Start(pi);
StreamReader streamReader = p.StandardOutput;
char[] buf = new char[256];
string line = string.Empty;
int count;
while ((count = streamReader.Read(buf, 0, 256)) > 0)
{
line += new String(buf, 0, count);
}
It works for only some cases.
The file that does not work has a size of 1.30 mb,
I don't know if that is the reason for it not working correctly.
line returns an empty string.
I hope this is clear.
Can someone point out my error? Thanks in advance.
A couple thoughts:
The various Read* methods of streamreader require you to ensure that your app has completed before they run, otherwise you may get no output depending on timing issues. You may want to look at the Process.WaitForExit() function if you want to use this route.
Also, unless you have a specific reason for allocating buffers (pain in the butt IMO) I would just use readline() in a loop, or since the process has exited, ReadToEnd() to get the whole output. Neither requires you to have to do arrays of char, which opens you up to math errors with buffer sizes.
If you want to go asynchronous and dump output as you run, you will want to use the BeginOutputReadLine() function (see MSDN)
Don't forget that errors are handled differently, so if for any reason your app is writing to STDERR, you will want to use the appropriate error output functions to read that output as well.
I have a (C#) console application which maintains a state. The state can be altered by feeding the application with various input through the console. I need to be able to both feed the application with a bit of input, then read the output rinse and repeat.
I create a new process and do all of the normal work of redirecting the input/output. The problem is that after I've sent input and call ReadLine() on the standard output it does not return a value before I call Close() on the standard input after which I cannot write anymore to the input stream.
How can I keep open the input stream while still receiving output?
var process = new Process
{
StartInfo =
{
FileName =
#"blabal.exe",
RedirectStandardInput = true,
RedirectStandardError = true,
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true,
ErrorDialog = false
}
};
process.EnableRaisingEvents = false;
process.Start();
var standardInput = process.StandardInput;
standardInput.AutoFlush = true;
var standardOutput = process.StandardOutput;
var standardError = process.StandardError;
standardInput.Write("ready");
standardInput.Close(); // <-- output doesn't arrive before after this line
var outputData = standardOutput.ReadLine();
process.Close();
process.Dispose();
The console application I'm redirecting IO from is very simple. It reads from the console using Console.Read() and writes to it using Console.Write(). I know for certain that this data is readable, since I have another application that reads from it using standard output / input (not written in .NET).
That is happening because of you are using Write("ready") which is will append a string to the text, instead use WriteLine("ready"). that simple :).
In C# (.NET 4.0 running under Mono 2.8 on SuSE) I would like to run an external batch command and capture its ouput in binary form. The external tool I use is called 'samtools' (samtools.sourceforge.net) and among other things it can return records from an indexed binary file format called BAM.
I use Process.Start to run the external command, and I know that I can capture its output by redirecting Process.StandardOutput. The problem is, that's a text stream with an encoding, so it doesn't give me access to the raw bytes of the output. The almost-working solution I found is to access the underlying stream.
Here's my code:
Process cmdProcess = new Process();
ProcessStartInfo cmdStartInfo = new ProcessStartInfo();
cmdStartInfo.FileName = "samtools";
cmdStartInfo.RedirectStandardError = true;
cmdStartInfo.RedirectStandardOutput = true;
cmdStartInfo.RedirectStandardInput = false;
cmdStartInfo.UseShellExecute = false;
cmdStartInfo.CreateNoWindow = true;
cmdStartInfo.Arguments = "view -u " + BamFileName + " " + chromosome + ":" + start + "-" + end;
cmdProcess.EnableRaisingEvents = true;
cmdProcess.StartInfo = cmdStartInfo;
cmdProcess.Start();
// Prepare to read each alignment (binary)
var br = new BinaryReader(cmdProcess.StandardOutput.BaseStream);
while (!cmdProcess.StandardOutput.EndOfStream)
{
// Consume the initial, undocumented BAM data
br.ReadBytes(23);
// ... more parsing follows
But when I run this, the first 23bytes that I read are not the first 23 bytes in the ouput, but rather somewhere several hundred or thousand bytes downstream. I assume that StreamReader does some buffering and so the underlying stream is already advanced say 4K into the output. The underlying stream does not support seeking back to the start.
And I'm stuck here. Does anyone have a working solution for running an external command and capturing its stdout in binary form? The ouput may be very large so I would like to stream it.
Any help appreciated.
By the way, my current workaround is to have samtools return the records in text format, then parse those, but this is pretty slow and I'm hoping to speed things up by using the binary format directly.
Using StandardOutput.BaseStream is the correct approach, but you must not use any other property or method of cmdProcess.StandardOutput. For example, accessing cmdProcess.StandardOutput.EndOfStream will cause the StreamReader for StandardOutput to read part of the stream, removing the data you want to access.
Instead, simply read and parse the data from br (assuming you know how to parse the data, and won't read past the end of stream, or are willing to catch an EndOfStreamException). Alternatively, if you don't know how big the data is, use Stream.CopyTo to copy the entire standard output stream to a new file or memory stream.
Since you explicitly specified running on Suse linux and mono, you can work around the problem by using native unix calls to create the redirection and read from the stream. Such as:
using System;
using System.Diagnostics;
using System.IO;
using Mono.Unix;
class Test
{
public static void Main()
{
int reading, writing;
Mono.Unix.Native.Syscall.pipe(out reading, out writing);
int stdout = Mono.Unix.Native.Syscall.dup(1);
Mono.Unix.Native.Syscall.dup2(writing, 1);
Mono.Unix.Native.Syscall.close(writing);
Process cmdProcess = new Process();
ProcessStartInfo cmdStartInfo = new ProcessStartInfo();
cmdStartInfo.FileName = "cat";
cmdStartInfo.CreateNoWindow = true;
cmdStartInfo.Arguments = "test.exe";
cmdProcess.StartInfo = cmdStartInfo;
cmdProcess.Start();
Mono.Unix.Native.Syscall.dup2(stdout, 1);
Mono.Unix.Native.Syscall.close(stdout);
Stream s = new UnixStream(reading);
byte[] buf = new byte[1024];
int bytes = 0;
int current;
while((current = s.Read(buf, 0, buf.Length)) > 0)
{
bytes += current;
}
Mono.Unix.Native.Syscall.close(reading);
Console.WriteLine("{0} bytes read", bytes);
}
}
Under unix, file descriptors are inherited by child processes unless marked otherwise (close on exec). So, to redirect stdout of a child, all you need to do is change the file descriptor #1 in the parent process before calling exec. Unix also provides a handy thing called a pipe which is a unidirectional communication channel, with two file descriptors representing the two endpoints. For duplicating file descriptors, you can use dup or dup2 both of which create an equivalent copy of a descriptor, but dup returns a new descriptor allocated by the system and dup2 places the copy in a specific target (closing it if necessary). What the above code does, then:
Creates a pipe with endpoints reading and writing
Saves a copy of the current stdout descriptor
Assigns the pipe's write endpoint to stdout and closes the original
Starts the child process so it inherits stdout connected to the write endpoint of the pipe
Restores the saved stdout
Reads from the reading endpoint of the pipe by wrapping it in a UnixStream
Note, in native code, a process is usually started by a fork+exec pair, so the file descriptors can be modified in the child process itself, but before the new program is loaded. This managed version is not thread-safe as it has to temporarily modify the stdout of the parent process.
Since the code starts the child process without managed redirection, the .NET runtime does not change any descriptors or create any streams. So, the only reader of the child's output will be the user code, which uses a UnixStream to work around the StreamReader's encoding issue,
I checked out what's happening with reflector. It seems to me that StreamReader doesn't read until you call read on it. But it's created with a buffer size of 0x1000, so maybe it does. But luckily, until you actually read from it, you can safely get the buffered data out of it: it has a private field byte[] byteBuffer, and two integer fields, byteLen and bytePos, the first means how many bytes are in the buffer, the second means how many have you consumed, should be zero. So first read this buffer with reflection, then create the BinaryReader.
Maybe you can try like this:
public class ThirdExe
{
private static TongueSvr _instance = null;
private Diagnostics.Process _process = null;
private Stream _messageStream;
private byte[] _recvBuff = new byte[65536];
private int _recvBuffLen;
private Queue<TonguePb.Msg> _msgQueue = new Queue<TonguePb.Msg>();
void StartProcess()
{
try
{
_process = new Diagnostics.Process();
_process.EnableRaisingEvents = false;
_process.StartInfo.FileName = "d:/code/boot/tongueerl_d.exe"; // Your exe
_process.StartInfo.UseShellExecute = false;
_process.StartInfo.CreateNoWindow = true;
_process.StartInfo.RedirectStandardOutput = true;
_process.StartInfo.RedirectStandardInput = true;
_process.StartInfo.RedirectStandardError = true;
_process.ErrorDataReceived += new Diagnostics.DataReceivedEventHandler(ErrorReceived);
_process.Exited += new EventHandler(OnProcessExit);
_process.Start();
_messageStream = _process.StandardInput.BaseStream;
_process.BeginErrorReadLine();
AsyncRead();
}
catch (Exception e)
{
Debug.LogError("Unable to launch app: " + e.Message);
}
private void AsyncRead()
{
_process.StandardOutput.BaseStream.BeginRead(_recvBuff, 0, _recvBuff.Length
, new AsyncCallback(DataReceived), null);
}
void DataReceived(IAsyncResult asyncResult)
{
int nread = _process.StandardOutput.BaseStream.EndRead(asyncResult);
if (nread == 0)
{
Debug.Log("process read finished"); // process exit
return;
}
_recvBuffLen += nread;
Debug.LogFormat("recv data size.{0} remain.{1}", nread, _recvBuffLen);
ParseMsg();
AsyncRead();
}
void ParseMsg()
{
if (_recvBuffLen < 4)
{
return;
}
int len = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(_recvBuff, 0));
if (len > _recvBuffLen - 4)
{
Debug.LogFormat("current call can't parse the NetMsg for data incomplete");
return;
}
TonguePb.Msg msg = TonguePb.Msg.Parser.ParseFrom(_recvBuff, 4, len);
Debug.LogFormat("recv msg count.{1}:\n {0} ", msg.ToString(), _msgQueue.Count + 1);
_recvBuffLen -= len + 4;
_msgQueue.Enqueue(msg);
}
The key is _process.StandardOutput.BaseStream.BeginRead(_recvBuff, 0, _recvBuff.Length, new AsyncCallback(DataReceived), null); and the very very important is that convert to asynchronous reads event like Process.OutputDataReceived.