In C# (.NET 4.0 running under Mono 2.8 on SuSE) I would like to run an external batch command and capture its ouput in binary form. The external tool I use is called 'samtools' (samtools.sourceforge.net) and among other things it can return records from an indexed binary file format called BAM.
I use Process.Start to run the external command, and I know that I can capture its output by redirecting Process.StandardOutput. The problem is, that's a text stream with an encoding, so it doesn't give me access to the raw bytes of the output. The almost-working solution I found is to access the underlying stream.
Here's my code:
Process cmdProcess = new Process();
ProcessStartInfo cmdStartInfo = new ProcessStartInfo();
cmdStartInfo.FileName = "samtools";
cmdStartInfo.RedirectStandardError = true;
cmdStartInfo.RedirectStandardOutput = true;
cmdStartInfo.RedirectStandardInput = false;
cmdStartInfo.UseShellExecute = false;
cmdStartInfo.CreateNoWindow = true;
cmdStartInfo.Arguments = "view -u " + BamFileName + " " + chromosome + ":" + start + "-" + end;
cmdProcess.EnableRaisingEvents = true;
cmdProcess.StartInfo = cmdStartInfo;
cmdProcess.Start();
// Prepare to read each alignment (binary)
var br = new BinaryReader(cmdProcess.StandardOutput.BaseStream);
while (!cmdProcess.StandardOutput.EndOfStream)
{
// Consume the initial, undocumented BAM data
br.ReadBytes(23);
// ... more parsing follows
But when I run this, the first 23bytes that I read are not the first 23 bytes in the ouput, but rather somewhere several hundred or thousand bytes downstream. I assume that StreamReader does some buffering and so the underlying stream is already advanced say 4K into the output. The underlying stream does not support seeking back to the start.
And I'm stuck here. Does anyone have a working solution for running an external command and capturing its stdout in binary form? The ouput may be very large so I would like to stream it.
Any help appreciated.
By the way, my current workaround is to have samtools return the records in text format, then parse those, but this is pretty slow and I'm hoping to speed things up by using the binary format directly.
Using StandardOutput.BaseStream is the correct approach, but you must not use any other property or method of cmdProcess.StandardOutput. For example, accessing cmdProcess.StandardOutput.EndOfStream will cause the StreamReader for StandardOutput to read part of the stream, removing the data you want to access.
Instead, simply read and parse the data from br (assuming you know how to parse the data, and won't read past the end of stream, or are willing to catch an EndOfStreamException). Alternatively, if you don't know how big the data is, use Stream.CopyTo to copy the entire standard output stream to a new file or memory stream.
Since you explicitly specified running on Suse linux and mono, you can work around the problem by using native unix calls to create the redirection and read from the stream. Such as:
using System;
using System.Diagnostics;
using System.IO;
using Mono.Unix;
class Test
{
public static void Main()
{
int reading, writing;
Mono.Unix.Native.Syscall.pipe(out reading, out writing);
int stdout = Mono.Unix.Native.Syscall.dup(1);
Mono.Unix.Native.Syscall.dup2(writing, 1);
Mono.Unix.Native.Syscall.close(writing);
Process cmdProcess = new Process();
ProcessStartInfo cmdStartInfo = new ProcessStartInfo();
cmdStartInfo.FileName = "cat";
cmdStartInfo.CreateNoWindow = true;
cmdStartInfo.Arguments = "test.exe";
cmdProcess.StartInfo = cmdStartInfo;
cmdProcess.Start();
Mono.Unix.Native.Syscall.dup2(stdout, 1);
Mono.Unix.Native.Syscall.close(stdout);
Stream s = new UnixStream(reading);
byte[] buf = new byte[1024];
int bytes = 0;
int current;
while((current = s.Read(buf, 0, buf.Length)) > 0)
{
bytes += current;
}
Mono.Unix.Native.Syscall.close(reading);
Console.WriteLine("{0} bytes read", bytes);
}
}
Under unix, file descriptors are inherited by child processes unless marked otherwise (close on exec). So, to redirect stdout of a child, all you need to do is change the file descriptor #1 in the parent process before calling exec. Unix also provides a handy thing called a pipe which is a unidirectional communication channel, with two file descriptors representing the two endpoints. For duplicating file descriptors, you can use dup or dup2 both of which create an equivalent copy of a descriptor, but dup returns a new descriptor allocated by the system and dup2 places the copy in a specific target (closing it if necessary). What the above code does, then:
Creates a pipe with endpoints reading and writing
Saves a copy of the current stdout descriptor
Assigns the pipe's write endpoint to stdout and closes the original
Starts the child process so it inherits stdout connected to the write endpoint of the pipe
Restores the saved stdout
Reads from the reading endpoint of the pipe by wrapping it in a UnixStream
Note, in native code, a process is usually started by a fork+exec pair, so the file descriptors can be modified in the child process itself, but before the new program is loaded. This managed version is not thread-safe as it has to temporarily modify the stdout of the parent process.
Since the code starts the child process without managed redirection, the .NET runtime does not change any descriptors or create any streams. So, the only reader of the child's output will be the user code, which uses a UnixStream to work around the StreamReader's encoding issue,
I checked out what's happening with reflector. It seems to me that StreamReader doesn't read until you call read on it. But it's created with a buffer size of 0x1000, so maybe it does. But luckily, until you actually read from it, you can safely get the buffered data out of it: it has a private field byte[] byteBuffer, and two integer fields, byteLen and bytePos, the first means how many bytes are in the buffer, the second means how many have you consumed, should be zero. So first read this buffer with reflection, then create the BinaryReader.
Maybe you can try like this:
public class ThirdExe
{
private static TongueSvr _instance = null;
private Diagnostics.Process _process = null;
private Stream _messageStream;
private byte[] _recvBuff = new byte[65536];
private int _recvBuffLen;
private Queue<TonguePb.Msg> _msgQueue = new Queue<TonguePb.Msg>();
void StartProcess()
{
try
{
_process = new Diagnostics.Process();
_process.EnableRaisingEvents = false;
_process.StartInfo.FileName = "d:/code/boot/tongueerl_d.exe"; // Your exe
_process.StartInfo.UseShellExecute = false;
_process.StartInfo.CreateNoWindow = true;
_process.StartInfo.RedirectStandardOutput = true;
_process.StartInfo.RedirectStandardInput = true;
_process.StartInfo.RedirectStandardError = true;
_process.ErrorDataReceived += new Diagnostics.DataReceivedEventHandler(ErrorReceived);
_process.Exited += new EventHandler(OnProcessExit);
_process.Start();
_messageStream = _process.StandardInput.BaseStream;
_process.BeginErrorReadLine();
AsyncRead();
}
catch (Exception e)
{
Debug.LogError("Unable to launch app: " + e.Message);
}
private void AsyncRead()
{
_process.StandardOutput.BaseStream.BeginRead(_recvBuff, 0, _recvBuff.Length
, new AsyncCallback(DataReceived), null);
}
void DataReceived(IAsyncResult asyncResult)
{
int nread = _process.StandardOutput.BaseStream.EndRead(asyncResult);
if (nread == 0)
{
Debug.Log("process read finished"); // process exit
return;
}
_recvBuffLen += nread;
Debug.LogFormat("recv data size.{0} remain.{1}", nread, _recvBuffLen);
ParseMsg();
AsyncRead();
}
void ParseMsg()
{
if (_recvBuffLen < 4)
{
return;
}
int len = IPAddress.NetworkToHostOrder(BitConverter.ToInt32(_recvBuff, 0));
if (len > _recvBuffLen - 4)
{
Debug.LogFormat("current call can't parse the NetMsg for data incomplete");
return;
}
TonguePb.Msg msg = TonguePb.Msg.Parser.ParseFrom(_recvBuff, 4, len);
Debug.LogFormat("recv msg count.{1}:\n {0} ", msg.ToString(), _msgQueue.Count + 1);
_recvBuffLen -= len + 4;
_msgQueue.Enqueue(msg);
}
The key is _process.StandardOutput.BaseStream.BeginRead(_recvBuff, 0, _recvBuff.Length, new AsyncCallback(DataReceived), null); and the very very important is that convert to asynchronous reads event like Process.OutputDataReceived.
Related
I'm trying to use NAudio to record some sound in C# using WasapiLoopbackCapture and WaveFileWriter.
The problem is that after the recording is finished, the "size" field in the WAV/RIFF header is set to 0, rendering the file unplayable.
I'm using the following code:
WasapiLoopbackCapture CaptureInstance = null;
WaveFileWriter RecordedAudioWriter = null;
void StartSoundRecord()
{
string outputFilePath = #"C:\RecordedSound.wav";
// Redefine the capturer instance with a new instance of the LoopbackCapture class
CaptureInstance = new WasapiLoopbackCapture();
// Redefine the audio writer instance with the given configuration
RecordedAudioWriter = new WaveFileWriter(outputFilePath, CaptureInstance.WaveFormat);
// When the capturer receives audio, start writing the buffer into the mentioned file
CaptureInstance.DataAvailable += (s, a) =>
{
// Write buffer into the file of the writer instance
RecordedAudioWriter.Write(a.Buffer, 0, a.BytesRecorded);
};
// When the Capturer Stops, dispose instances of the capturer and writer
CaptureInstance.RecordingStopped += (s, a) =>
{
RecordedAudioWriter.Dispose();
RecordedAudioWriter = null;
CaptureInstance.Dispose();
};
// Start audio recording !
CaptureInstance.StartRecording();
}
void StopSoundRecord()
{
if(CaptureInstance != null)
{
CaptureInstance.StopRecording();
}
}
(Borrowed from: https://ourcodeworld.com/articles/read/702/how-to-record-the-audio-from-the-sound-card-system-audio-with-c-using-naudio-in-winforms )
Which I'm testing with simply:
StartSoundRecord();
Thread.Sleep(10000);
StopSoundRecord();
What am I missing, why isn't WaveFileWriter writing the size field? I have tried also calling the Flush() and Close() methods before disposing. But it makes no difference.
Sure, I could write a method to find out the size of the file and manually writing it to the final file, but that seems unnecessary.
Found the solution.
Calling RecordedAudioWriter.Flush() after every Write made it work just fine.
Don't know if doing so might be inefficient (as I assume flush is blocking until data is written to disk), but for my application that is not an issue.
I have two applications:
WPF that is a client
Console Application that is WCF service hosted stand-alone
I want to be able to start process through WCF and return an output stream from that process (so I will use process.StandardOutput.BaseStream).
The problem is that once process starts, the returning object which is output stream is not being returned until process exits.
TransferMode in my both programatically created bindings is set to Streamed.
Uploading files TO WCF works fine, but problem exists only with returning stream.
WCF Service Code:
[OperationContract]
Stream RunTests(string testPackageDll);
Implementation:
public Stream RunTests(string testPackageDll)
{
var p = new Process()
{
StartInfo =
{
FileName = #"C:\Tests\NUnit-2.6.4\nunit-console.exe",
Arguments = $#"C:\Tests\{testPackageDll} /xml=test-results.xml",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
}
};
try
{
p.Start();
p.BeginOutputReadLine();
return p.StandardOutput.BaseStream;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
return null;
}
}
WPF Application Code:
public void RunTests(string testToRun)
{
string streamLine;
using (TextReader reader = new StreamReader(_serviceHandler.Service.RunTests(testToRun)))
{
streamLine = reader.ReadLine();
while (streamLine != null)
{
WriteToConsole(streamLine);
streamLine = reader.ReadLine();
}
}
}
the _serviceHandler.Service.RunTests(testToRun) method does not return Stream as it suppose to do, instead it is waiting for process to end then returns stream.
What I want to achieve is to read "live" stream straight from process.
UPDATE
I just found out that Stream is actually being streamed through WCF but only when the output text buffer reaches 12-16 kb, so what I am looking for now is how to change frequency (probably buffer size if there's a chance) of the data.
I know it would be possible by "chunking" the stream into small parts, but maybe there's some better solution than that?
I am using NAudio library to record systems mic input - continuously.
private void RecordStart() {
try {
_sourceStream = new WaveIn {
DeviceNumber = _recordingInstance.InputDeviceIndex,
WaveFormat =
new WaveFormat(
44100,
WaveIn.GetCapabilities(_recordingInstance.InputDeviceIndex).Channels)
};
_sourceStream.DataAvailable += SourceStreamDataAvailable;
_sourceStream.StartRecording();
} catch (Exception exception) {
Log.Error("Recording failes", exception);
}
}
there is an event handler which will get the data from recording stream, whenever data is available.
I was able to create an audio (mp3) HTTP streaming, with the VLC player installed in my system - with an existing audio file.
const int portNumber = 8089;
const string streamName = "fstream_1789846";
const string audio = "C:\\Recording\\Audio\\1789846.wav";
const string windowQuiet = "-I dummy --dummy-quiet";
const string tanscode = ":sout=#transcode{vcodec=none,acodec=mp3,ab=128,channels=2,samplerate=44100}";
var stream = String.Format(#":http{{mux=mp3,dst=:{0}/{1}}}", portNumber, streamName);
const string keep = ":sout-keep";
var vlcStreamParamList = new List<string> {windowQuiet, audio, tanscode+stream, keep};
var process = new Process
{
StartInfo =
{
FileName = #"C:\Program Files (x86)\VideoLAN\VLC\vlc.exe",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
}
};
var vlcParamString = String.Join(" ", vlcStreamParamList);
process.StartInfo.Arguments = vlcParamString;
process.Start();
In the above code, I have a recording completed final audio file in the source directory - C:\Recording\Audio\1789846.wav
I have tried to create the stream with the active file - which is continuously being updated by a recorder. (SourceStreamDataAvailable function write bytes to this file). I used same code above.
It stops streaming after few seconds/minutes. (I think it is creating a stream from the initially available data only, and it is not using the updated file/content.)
Is there any way to solve this problem??
Is it possible to create an HTTP audio stream with the bytes I have captured from the mic using NAudio recorder.? without creating a file.
Within the SourceStreamDataAvailable function, sent whatever content is available to the stream, and keep updating it.
private void SourceStreamDataAvailable(object sender, WaveInEventArgs e) {
//create an http stream with Vlc/LibVlc/BASS.Net or any library
//and stream the bytes received from recording.
//e.BytesRecorded);
}
I have done enough googling to implement this solution but no luck.
I am okay with LibVlc or any VLC open source libraries for c#.
I have tried calling a Process(console application) using the following code:
ProcessStartInfo pi = new ProcessStartInfo();
pi.UseShellExecute = false;
pi.RedirectStandardOutput = true;
pi.CreateNoWindow = true;
pi.FileName = #"C:\fakepath\go.exe";
pi.Arguments = "FOO BAA";
Process p = Process.Start(pi);
StreamReader streamReader = p.StandardOutput;
char[] buf = new char[256];
string line = string.Empty;
int count;
while ((count = streamReader.Read(buf, 0, 256)) > 0)
{
line += new String(buf, 0, count);
}
It works for only some cases.
The file that does not work has a size of 1.30 mb,
I don't know if that is the reason for it not working correctly.
line returns an empty string.
I hope this is clear.
Can someone point out my error? Thanks in advance.
A couple thoughts:
The various Read* methods of streamreader require you to ensure that your app has completed before they run, otherwise you may get no output depending on timing issues. You may want to look at the Process.WaitForExit() function if you want to use this route.
Also, unless you have a specific reason for allocating buffers (pain in the butt IMO) I would just use readline() in a loop, or since the process has exited, ReadToEnd() to get the whole output. Neither requires you to have to do arrays of char, which opens you up to math errors with buffer sizes.
If you want to go asynchronous and dump output as you run, you will want to use the BeginOutputReadLine() function (see MSDN)
Don't forget that errors are handled differently, so if for any reason your app is writing to STDERR, you will want to use the appropriate error output functions to read that output as well.
I once again need your help figuring out this problem of mine...Been already a day and I can't seem to find out why this is happening in my code and output.
Ok.....so basically I am trying to implement the RCON Protocol of Valve in C#, so far I am getting the expected output given the code and sample usage below:
Usage:
RconExec(socket, "cvarlist");
Code:
private string RconExec(Socket sock, string command)
{
if (!sock.Connected) throw new Exception("Not connected");
//sock.DontFragment = true;
sock.ReceiveTimeout = 10000;
sock.SendTimeout = 10000;
//sock.Blocking = true;
Debug.WriteLine("Executing RCON Command: " + command);
byte[] rconCmdPacket = GetRconCmdPacket(command);
sock.Send(rconCmdPacket); //Send the request packet
sock.Send(GetRconCmdPacket("echo END")); //This is the last response to be received from the server to indicate the end of receiving process
RconPacket rconCmdResponsePacket = null;
string data = null;
StringBuilder cmdResponse = new StringBuilder();
RconPacket packet = null;
int totalBytesRead = 0;
do
{
byte[] buffer = new byte[4]; //Allocate buffer for the packet size field
int bytesReceived = sock.Receive(buffer); //Read the first 4 bytes to determine the packet size
int packetSize = BitConverter.ToInt32(buffer, 0); //Get the packet size
//Now proceed with the rest of the data
byte[] responseBuffer = new byte[packetSize];
//Receive more data from server
int bytesRead = sock.Receive(responseBuffer);
//Parse the packet by wrapping under RconPacket class
packet = new RconPacket(responseBuffer);
totalBytesRead += packet.String1.Length;
string response = packet.String1;
cmdResponse.Append(packet.String1);
Debug.WriteLine(response);
Thread.Sleep(50);
} while (!packet.String1.Substring(0,3).Equals("END"));
Debug.WriteLine("DONE..Exited the Loop");
Debug.WriteLine("Bytes Read: " + totalBytesRead + ", Buffer Length: " + cmdResponse.Length);
sock.Disconnect(true);
return "";
}
The Problem:
This is not yet the final code as I am just testing the output in the Debug window. There are a couple of issues occuring if I modify the code to it's actual state.
Removing Thread.Sleep(50)
If I remove Thread.Sleep(50), the output doesn't complete and ends up throwing an exception. I noticed the 'END' termination string is sent by the server pre-maturely. This string was expected to be sent by the server only when the whole list completes.
I tested this numerous times and same thing happens, if I don't remove the line, the list completes and function exits the loop properly.
Removing Debug.WriteLine(response); within the loop and outputting the string using Debug.WriteLine(cmdResponse.ToString()); outside the loop, only partial list data is displayed. If I compare the actual bytes read from the loop with the length of the StringBuilder instance, they're just the same? Click here for the output generated.
Why is this happening given the two scenarios mentioned above?
You are not considering that Socket.Receive very well could read fewer bytes than the length of the supplied buffer. The return value tells you the number of bytes that was actually read. I see that you are properly storing this value in a variable, but I cannot see any code that use it.
You should be prepared to make several calls to Receive to retrieve the entire package. In particular when you receive the package data.
I'm not sure that this is the reason for your problem. But it could be, since a short delay on the client side could be enough to fill the network buffers so that the entire package is read in a single call.
Try using the following code to retrieve package data:
int bufferPos = 0;
while (bufferPos < responseBuffer.Length)
{
bufferPos += socket.Receive(responseBuffer, bufferPos, responseBuffer.Length - bufferPos, SocketFlags.None);
}
Note: You should also support the case when the first call to Receive (the one where you receive the package's data length) doesn't return 4 bytes.