I was executing BCP command through exe and it stucks after copying 50000 rows.
I looked at some forums and got to know that if we use StandardOuputReader in code than its max limit of output is near to 50000 rows which is happening to me as well
Is there a way i ran redirect output where more than 50000 rows can get out.
This code works here i have
proc.StartInfo.RedirectStandardOutput = false;
but i want to have it as true, to see the outputs.
private static void RunBatch(string Fullfilepath, string BatchFilePathDumpFlatFile)
{
mLogger.Error("RunBatch Start=== >");
mLogger.Error("Batch Filepath " + Fullfilepath + '\n' + "Batch File Directory " + BatchFilePathDumpFlatFile);
Process proc = null;
string targetDir = BatchFilePathDumpFlatFile;
proc = new Process();
proc.StartInfo.WorkingDirectory = targetDir;
proc.StartInfo.FileName = Fullfilepath;
//proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.Arguments = "/c";
proc.StartInfo.RedirectStandardError = true;
proc.StartInfo.RedirectStandardOutput = false;
proc.Start();
proc.WaitForExit();
string output = proc.StandardOutput.ReadToEnd();
proc.WaitForExit();
string error = proc.StandardError.ReadToEnd();
proc.WaitForExit();
mLogger.Error("Output from batch " + output);
mLogger.Error("Error From Batch " + error);
}
update 1:
private static void RunBatch(string Fullfilepath, string BatchFilePathDumpFlatFile)
{
mLogger.Error("RunBatch Start=== >");
mLogger.Error("Batch Filepath " + Fullfilepath + '\n' + "Batch File Directory " + BatchFilePathDumpFlatFile);
Process proc = null;
string targetDir = BatchFilePathDumpFlatFile;
proc = new Process();
proc.StartInfo.WorkingDirectory = targetDir;
proc.StartInfo.FileName = Fullfilepath;
//proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.Arguments = "/c";
proc.StartInfo.RedirectStandardError = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.Start();
string output = proc.StandardOutput.ReadToEnd();
proc.WaitForExit();
string error = proc.StandardError.ReadToEnd();
proc.WaitForExit();
mLogger.Error("Output from batch " + output);
mLogger.Error("Error From Batch " + error);
}
this is what i am using is there a mistake because still BCP hangs and it starts running when i stop the exe of the code.
This is the classic deadlock condition. You should not call WaitForExit before reading the StandardOutput fully.
When the output is redirected, process will not terminate before all of its StandardOutput stream is read. So calling WaitForExit will wait for the started process to terminate, and child process will wait for the parent process to read the output stream fully before it can finish and thus deadlocks.
Msdn provides the explanation and code sample to avoid deadlock.
Synchronous read operations introduce a dependency between the caller reading from the StandardOutput stream and the child process writing to that stream. These dependencies can result in deadlock conditions. When the caller reads from the redirected stream of a child process, it is dependent on the child. The caller waits on the read operation until the child writes to the stream or closes the stream. When the child process writes enough data to fill its redirected stream, it is dependent on the parent. The child process waits on the next write operation until the parent reads from the full stream or closes the stream. The deadlock condition results when the caller and child process wait on each other to complete an operation, and neither can proceed. You can avoid deadlocks by evaluating dependencies between the caller and child process.
// Start the child process.
Process p = new Process();
// Redirect the output stream of the child process.
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = "Write500Lines.exe";
p.Start();
// Do not wait for the child process to exit before
// reading to the end of its redirected stream.
// p.WaitForExit();
// Read the output stream first and then wait.
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
The code example avoids a deadlock condition by calling p.StandardOutput.ReadToEnd before p.WaitForExit. A deadlock condition can result if the parent process calls p.WaitForExit before p.StandardOutput.ReadToEnd and the child process writes enough text to fill the redirected stream. The parent process would wait indefinitely for the child process to exit. The child process would wait indefinitely for the parent to read from the full StandardOutput stream.
There is a similar issue when you read all text from both the standard output and standard error streams. The following C# code, for example, performs a read operation on both streams.
// Do not perform a synchronous read to the end of both
// redirected streams.
// string output = p.StandardOutput.ReadToEnd();
// string error = p.StandardError.ReadToEnd();
// p.WaitForExit();
// Use asynchronous read operations on at least one of the streams.
p.BeginOutputReadLine();
string error = p.StandardError.ReadToEnd();
p.WaitForExit();
Almost everything above taken from msdn, I suggest you read it fully to avoid creating deadlocks further.
Related
I'm trying read stdout byte stream into MemoryStream. All's works properly until I run same code in parallel threads
using (Process proc = new Process())
{
proc.StartInfo.FileName = eac3toFullPath;
proc.StartInfo.Arguments = "input.flac stdout.wav";
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.Start();
proc.StandardOutput.BaseStream.CopyTo(mem);
proc.WaitForExit();
if (proc.ExitCode > 0)
{
throw new ConsoleAppException(proc);
}
mem.Position = 0;
}
Problem occurs when first thread execute stdout copy to mem and in the same time second thread report ExitCode equal '1'
I started console app (CLI) from WinForm application so where is no console window allocated due to CreateNoWindow = true;
So I repeat myself. The code works well, the ExitCode of CLI process is always zero,
until I run the code above multiple times simultaneously.
StandardOutput is redirected as of matter of fact I can't see any friendly error output. I also tried RedirectStandardError and ErrorDataReceived event and BeginErrorReadLine() with no luck.
I've been racking my brain trying to figure out why I can not use >> (append) as a part of my p.StartInfo.Argument. When I remove ">" it works perfectly, but the moment I try to use ">" or ">>" I get a "Incorrect Parameter" error. I've gone ahead and wrote my program without > or >> and just stored the output into a string and write it to a file later. Could someone explain why ">" or ">>" would not work in this situation?
// Start the child process.
Process p = new Process();
// Redirect the output stream of the child process.
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = "attrib.exe";
startInfo.Arguments = "/S *.jpg > mypics.txt";
p.Start();
The output redirection operator > is a feature of cmd.exe, not the operating system. The naive way to use it is hence to call cmd as so:
p.StartInfo.FileName = "cmd.exe";
startInfo.Arguments = "/C \"attrib.exe /S *.jpg > mypics.txt\"";
The proper way to redirect the output, however, is to first set StartInfo.RedirectStandardOutput to true (which you did), and then pipe the output from Process.StandardOutput to the file, as so:
using(StreamWriter file = new StreamWriter("mypics.txt")) {
p.StandardOutput.CopyTo(file);
}
Or, async version:
using(StreamWriter file = new StreamWriter("mypics.txt")) {
await p.StandardOutput.CopyToAsync(file);
}
That's because > and >> are not arguments interpreted by attrib.exe, they're instructions to cmd.exe.
You could try instead something like:
p.StartInfo.FileName = "cmd.exe";
p.StartInfo.Arguments = "/c attrib.exe /S *.jpg > mypics.txt";
Also, this is a red herring:
p.StartInfo.RedirectStandardOutput = true;
If you wanted to read the output in C# code and write the file yourself, you'd use this option. It's not helpful for using the > output redirector.
SlugFiller's solution is good but doesn't work reliably with large amounts of output. The problem is that if the process outputs too much text it hangs because the stream reaches the maximum buffer size. In other words the stream needs to be spooled out as the process runs, preventing the buffer from ever filling up.
However, there is a way to do this using a task to receive data from the output stream and spool it out in real-time.
// Start the child process.
Process p = new Process();
// Redirect the output stream of the child process.
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = "attrib.exe";
startInfo.Arguments = "/S *.jpg > mypics.txt";
p.Start();
Thread stdoutThread = new Thread(new ThreadStart(WriteStandardOutput));
stdoutThread.IsBackground = true;
stdoutThread.Name = "stdout_writer";
stdoutThread.Start();
private void WriteStandardOutput()
{
using (StreamWriter sw = File.CreateText(ConsoleLogFileFullPath))
using (StreamReader sr = p.StandardOutput)
{
for (;;)
{
string line = sr.ReadLine();
if (line == null)
break;
sw.WriteLine(textLine);
}
sw.Flush();
}
}
At some point be sure to call stdoutThread.Join() to make sure that the thread completes. I have tested this with some code that generates large amounts of output and it's still possible to overflow the buffer, or to break ReadLine() with extremely long lines, but it does work for very fast output in the order of a few thousand 80 character lines per second.
I am working on an application that calls several command line applications to do some post processing on some video files.
Right now I am trying to use Comskip to identify the commercial breaks in a video recording from my cable card tuner. This runs just fine, but I am having problems getting the screen output that I need.
String stdout = null;
using (var process = new Process())
{
var start = new ProcessStartInfo(comskip, cmdLine);
start.WindowStyle = ProcessWindowStyle.Normal;
start.CreateNoWindow = true;
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
process.StartInfo = start;
process.Start();
process.WaitForExit();
stdout = process.StandardOutput.ReadToEnd();
}
I'm expecting stdout to grab what is displayed on the screen the same as when the application is launched manually (screen shot below) which is a continuous feed of what the application is doing, and mixed in the output are lines that give a % progress, which I want to use to update a progress bar
But running the above code only gives me:
The commandline used was:
"C:\Users\Chris\Google Drive\Tools\ComSkip\comskip.exe" "C:\Users\Chris\Desktop\ComSkip Tuning Files\Modern Family.wtv" "--ini=C:\Users\Chris\Desktop\ComSkip Tuning Files\comskip_ModernFamily.ini"
Setting ini file to C:\Users\Chris\Desktop\ComSkip Tuning Files\comskip_ModernFamily.ini as per commandline
Using C:\Users\Chris\Desktop\ComSkip Tuning Files\comskip_ModernFamily.ini for initiation values.
I also tried redirecting the StandardError stream and grabbing process.StandardError.ReadToEnd(); but the process appears to hang if I run with these options.
Am I missing something to capture what I'm hoping for, or is it possible that the output stream for this application is going somewhere else that is not accessible?
You must set following:
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.UseShellExecute = false;
process.OutputDataReceived += new DataReceivedEventHandler(ReadOutput);
process.ErrorDataReceived += new DataReceivedEventHandler(ErrorOutput);
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
process.WaitForExit();
and catch the output in ReadOutput and ErrorOutput
private static void ErrorOutput(object sender, DataReceivedEventArgs e)
{
if (e.Data != null)
{
stdout = "Error: " + e.Data;
}
}
private static void ReadOutput(object sender, DataReceivedEventArgs e)
{
if (e.Data != null)
{
stdout = e.Data;
}
}
See the docs on RedirectStandardOutput. Waiting for the child process to end before reading the output can cause a hang.
It particular, the example says not to do what you have done:
Process p = new Process();
// Redirect the output stream of the child process.
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = "Write500Lines.exe";
p.Start();
// Do not wait for the child process to exit before
// reading to the end of its redirected stream.
// p.WaitForExit();
// Read the output stream first and then wait.
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
You should use the events OutputDataReceived and possibly ErrorDataReceived and update the progress bar in the handler.
I have a user, let's call it "MyUser". It has a password, suppose it is "Password". This user has an SSH key for git. I try to run from my ASP.NET application a batch file which issues git commands, it is at a location which is passed as a parameter. My function is as follows:
private void ExecuteCommand(string path, int timeout)
{
Process process = new Process();
process.StartInfo = new ProcessStartInfo();
process.StartInfo.FileName = "cmd.exe";
process.StartInfo.Arguments = "\"" + path + "\"";
process.StartInfo.CreateNoWindow = true;
process.StartInfo.UseShellExecute = false;
//processInfo.WorkingDirectory = Config.GitHubRepoPath;
process.StartInfo.UserName = "MyUser";
process.StartInfo.Password = new System.Security.SecureString();
process.StartInfo.Password.AppendChar('P');
process.StartInfo.Password.AppendChar('a');
process.StartInfo.Password.AppendChar('s');
process.StartInfo.Password.AppendChar('s');
process.StartInfo.Password.AppendChar('w');
process.StartInfo.Password.AppendChar('o');
process.StartInfo.Password.AppendChar('r');
process.StartInfo.Password.AppendChar('d');
// *** Redirect the output ***
process.StartInfo.RedirectStandardError = true;
process.StartInfo.RedirectStandardOutput = true;
process.Start();
// *** Read the streams ***
string output = process.StandardOutput.ReadToEnd();
string error = process.StandardError.ReadToEnd();
if (timeout <= 0)
{
process.WaitForExit();
}
else
{
process.WaitForExit(timeout);
}
int exitCode = process.ExitCode;
process.Close();
return new ShellCommandReturn { Error = error, ExitCode = exitCode, Output = output };
}
But when I run this function, the ExitCode is -1073741502 and error and output are empty. How can I fix this behavior?
Please help me, I have tried to solve this literally for days.
I think redirecting both standard error and standard output & attempts to consume both synchronously is wrong. Please see this link:
http://msdn.microsoft.com/en-us/library/system.diagnostics.processstartinfo.redirectstandardoutput%28v=vs.100%29.aspx
Allow me to copy excerpt:
A deadlock condition results if the parent process calls p.StandardOutput.ReadToEnd followed by p.StandardError.ReadToEnd and the child process writes enough text to fill its error stream. The parent process would wait indefinitely for the child process to close its StandardOutput stream. The child process would wait indefinitely for the parent to read from the full StandardError stream.
The other thing is ... when you invoke a cmd.exe instance, try adding a "/c" argument too.
I have to use an other application (console) to pass some parameter to this program and inside my C# program get the output of that program. I would like not to see the console (all invisible to the user). How can I do that?
Process myProcess = new Process();
ProcessStartInfo myProcessStartInfo = new ProcessStartInfo("YOUPROGRAM_CONSOLE.exe" );
myProcessStartInfo.UseShellExecute = false;
myProcessStartInfo.RedirectStandardOutput = true;
myProcess.StartInfo = myProcessStartInfo;
myProcess.Start();
StreamReader myStreamReader = myProcess.StandardOutput;
string myString = myStreamReader.ReadLine();
Console.WriteLine(myString);
myProcess.Close();
Source : MSDN
Edited:
If you require to get the Error Message you will need to use Async operation. You can use asynchronous read operations to avoid these dependencies and their deadlock potential. Alternately, you can avoid the deadlock condition by creating two threads and reading the output of each stream on a separate thread.