I want to process a large ammount of data stored in a text file. Here is the code I use to make it work faster:
var result = File
.ReadLines(textBox1.Text)
.AsParallel()
.WithDegreeOfParallelism(100)
.Select(line => ProcessLine(line));
The method ProcessLine gets the line then processes it and add it to an ArrayList.
After all the processing is done I load the ArrayList into a Datagrid,
but sometimes it completes all the lines and sometimes it hangs, I don't know why.
Any suggestions ?
Update
Here is the Method ProcessLine
private string ProcessLine(string domain)
{
ProcessStartInfo cmdinfo = new ProcessStartInfo();
cmdinfo.FileName = "cmd.exe";
cmdinfo.Arguments = "/c nslookup";
cmdinfo.RedirectStandardInput = true;
cmdinfo.RedirectStandardOutput = true;
cmdinfo.CreateNoWindow = true;
cmdinfo.UseShellExecute = false;
cmdinfo.RedirectStandardError = false;
Process cmdd = new Process();
cmdd = Process.Start(cmdinfo);
string spf = "none";
createproc:
try
{
cmdd.StandardInput.WriteLine("set q=txt");
cmdd.StandardInput.Flush();
cmdd.StandardInput.WriteLine(domain);
cmdd.StandardInput.WriteLine("exit");
cmdd.StandardInput.WriteLine("exit");
StreamReader r = cmdd.StandardOutput;
//cmdd.WaitForExit();
cmdd.Close();
spf = "";
string rdl = string.Empty;
bool spffound = false;
while (rdl != null)
{
try
{
rdl = r.ReadLine();
if (rdl.Contains("v=spf"))
{
spffound = true;
spf = rdl.Trim();
this.Invoke(new MethodInvoker(delegate
{
textBox2.AppendText("domain found : " + domain + Environment.NewLine + "SPF = " + spf + Environment.NewLine);
textBox2.Update();
}));
break;
}
}
catch (Exception)
{
}
}
if (!spffound)
spf = "none";
nbrDoms++;
this.Invoke(new MethodInvoker(delegate
{
DomsElapsed.Text = nbrDoms + " Domains Elapsed";
DomsElapsed.Update();
}));
SPFRecord srx = new SPFRecord((string)spf.Clone(), (string)domain.Clone());
if (srx == null)
{
cmdd.Kill();
cmdinfo = new ProcessStartInfo();
cmdinfo.FileName = "cmd.exe";
cmdinfo.Arguments = "/c nslookup";
cmdinfo.RedirectStandardInput = true;
cmdinfo.RedirectStandardOutput = true;
cmdinfo.CreateNoWindow = true;
cmdinfo.UseShellExecute = false;
cmdinfo.RedirectStandardError = false;
cmdd = new Process();
cmdd.StartInfo = cmdinfo;
cmdd.Start();
goto createproc;
}
lock (pageManager)
{
pageManager.AddRecord(srx);
}
//this.Invoke(new MethodInvoker(delegate
//{
//}));
}
catch(Exception exc)
{
cmd.Kill();
cmdinfo = new ProcessStartInfo();
cmdinfo.FileName = "cmd.exe";
cmdinfo.Arguments = "/c nslookup";
cmdinfo.RedirectStandardInput = true;
cmdinfo.RedirectStandardOutput = true;
cmdinfo.CreateNoWindow = true;
cmdinfo.UseShellExecute = false;
cmdinfo.RedirectStandardError = false;
cmdd = new Process();
cmdd.StartInfo = cmdinfo;
cmdd.Start();
Thread.Sleep(10);
goto createproc;
}
return "";
}
Read the lines of text into a string, with something like file.readalllines(psudo code)
Basically each thread is locking the other, are you trying this for speed or because the file is too large to fit into memory?
Ok, a few things to mention:
Do not use goto statements - it's hard to understand what your method does. Just move the creation of the Process into a separate method and call that method instead of using goto
Processes do take time to load and quite a lot for what you want to do. To avoid this load penalty, instead of creating and calling a process try to replace that with a method which does the same. There's an example of nslookup without calling the process. Try adapting it to your needs
Remove the locks - if your application somehow gets to use 100 threads the lock is a waste of time. You'll have 99 threads waiting for the other single thread to push its data to the pageManager. As #Mrinal Kamboj pointed out, you can use a thread-safe collection. In this case, use a BlockingCollection<T> and add the results there. At the other end of the queue have the pageManager listening and consuming each item as it arrives.
The UI needs its separate cycles to refresh which also takes time. If pageManager.AddRecord() somehow has to refresh the UI then the other threads won't wait just for the add operation.
UI updates must be done in the thread that created the controls and that thread can't update the UI if it's waiting for another thread.
The overall algorithm should look like this:
public class Engine
{
private readonly BlockingCollection<string> _messagePipeline = new BlockingCollection<string>();
public BlockingCollection<string> MessagePipeline
{
get { return _messagePipeline; }
}
public void Process(string file)
{
File.ReadLines(file)
.AsParallel()
.ForAll(line =>
{
var nsLookupResult = NsLookupMethod(line);
if(nsLookupResult.HasInfoYouNeed)
_messagePipeline.Add(nsLookupResult.DisplayInfo);
});
}
}
public class MainForm : Form
{
private readonly Engine _engine; // ...
private void OnStartButtonClick(object sender, EventArgs e)
{
var cts = new CancellationTokenSource();
_engine.Process(textbox1.Text);
Task.Factory.StartNew(()=>
{
foreach(var message in _engine.MessagePipeline.GetConsumingEnumerable())
{
// show the message
Application.DoEvents(); // allow the app to process other events not just pushing messages.
}
}, cts.Token,
TaskCreationOptions.PreferFairness,
// Specify that you want UI updates to be done on the UI thread
// and not on any other thread
TaskScheduler.FromCurrentSynchronizationContext());
}
}
And that should do it. I do have a (more or less academic) example of this kind of logic in action. The UI update logic is in the MainForm of the application and the processing logic is in an Engine class; have a look there.
I am working on a C# Form tool that will help me convert all of my phone and DSLR video to HEVC, currently, i have a program that uploads the photos and videos to different directories in my home server each time i connect to the WiFi. Once a month or so, i manually convert all the videos, but thought I would automate the process.. I have the Form working perfectly for processing 1 file. but get into trouble when processing a Directory (with possible sub-directories) all at once..
Sorry, this is long, just want to be thorough. here is the button calls
private void processFile_Click(object sender, EventArgs e)
{
OpenFileDialog file = new OpenFileDialog();
file.InitialDirectory = baseMediaDirectory;
if (file.ShowDialog() == DialogResult.OK)
{
ProcessSinlgeFile(file.FileName);
}
}
(above)for one file and (below) for a directory
private void processDirectory_Click(object sender, EventArgs e)
{
FolderBrowserDialog file = new FolderBrowserDialog();
file.SelectedPath = baseMediaDirectory;
if(file.ShowDialog() == DialogResult.OK)
{
ProcessDirectoryOfFiles(file.SelectedPath);
}
}
private void ProcessDirectoryOfFiles(string selectedPath)
{
List<string> listOfFiles = GetAllFiles(selectedPath);
foreach (string s in listOfFiles)
{
ProcessSinlgeFile(s);
}
}
both ultimately call this method, to do some checks and setup
private void ProcessSinlgeFile(string fileName)
{
if (IsAcceptableMediaFile(fileName))
{
outputWindow.AppendText("File to Process: " + fileName);
processMediaFile =
new MediaFileWrapper(this.outputWindow, new MediaFile(fileName), new NReco.VideoInfo.FFProbe());
if (processMediaFile.OkToProcess)
{
int initialCRFValue = 15;
//ultrafast superfast veryfast faster fast medium slow slower veryslow placebo
string intialSpeed = "veryfast";
try {
ConvertToMPEG(processMediaFile.getFFMPEGCommand(initialCRFValue, intialSpeed), processMediaFile);
}
catch
{
// at somepoint, we'll catch a bad file size (or compression)
// then change the CRF value and/or compression speed
}
}
}
}
ultimately I get to this Method and run into trouble.
private async void ConvertToMPEG(string arguments, MediaFileWrapper processMediaFile)
{
startTime = DateTime.Now;
watch = new Stopwatch();
watch.Start();
progressBar1.Minimum = 0;
progressBar1.Maximum = processMediaFile.GetTotalMilliseconds();
// Start the child process.
p = new Process();
//Setup filename and arguments
outputWindow.AppendText("ffmpeg " + arguments);
p.StartInfo.Arguments = arguments;
p.StartInfo.FileName = "ffmpeg.exe";
p.StartInfo.UseShellExecute = false;
// Redirect the output stream of the child process.
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.RedirectStandardInput = true;
// capture the date for stdout and std error
// note FFMPEG uses Stderr exclusively
p.ErrorDataReceived += new DataReceivedEventHandler(ErrorDataReceived);
p.OutputDataReceived += new DataReceivedEventHandler(OutputDataReceived);
// Hide Console Window
p.StartInfo.CreateNoWindow = true;
p.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
p.Start();
p.BeginErrorReadLine();
p.BeginOutputReadLine();
await p.WaitForExitAsync();
}
and WaitForExitAsync is in another class because in can not be in here with a Form
public static Task WaitForExitAsync(this Process process,
CancellationToken cancellationToken = default(CancellationToken))
{
var tcs = new TaskCompletionSource<object>();
process.EnableRaisingEvents = true;
process.Exited += (sender, args) => tcs.TrySetResult(null);
if (cancellationToken != default(CancellationToken))
cancellationToken.Register(tcs.SetCanceled);
return tcs.Task;
}
however, single files work fine, when I call a directory through, it continuously starts processes for each file, trying to run them all at the same time. You can see I tried implementing this
process.WaitForExit() asynchronously
with no luck.
I have the following code:
info = new System.Diagnostics.ProcessStartInfo("TheProgram.exe", String.Join(" ", args));
info.CreateNoWindow = true;
info.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
info.RedirectStandardOutput = true;
info.UseShellExecute = false;
System.Diagnostics.Process p = System.Diagnostics.Process.Start(info);
p.WaitForExit();
Console.WriteLine(p.StandardOutput.ReadToEnd()); //need the StandardOutput contents
I know that the output from the process I am starting is around 7MB long. Running it in the Windows console works fine. Unfortunately programmatically this hangs indefinitely at WaitForExit. Note also this code does NOT hang for smaller outputs (like 3KB).
Is it possible that the internal StandardOutput in ProcessStartInfo can't buffer 7MB? If so, what should I do instead? If not, what am I doing wrong?
The problem is that if you redirect StandardOutput and/or StandardError the internal buffer can become full. Whatever order you use, there can be a problem:
If you wait for the process to exit before reading StandardOutput the process can block trying to write to it, so the process never ends.
If you read from StandardOutput using ReadToEnd then your process can block if the process never closes StandardOutput (for example if it never terminates, or if it is blocked writing to StandardError).
The solution is to use asynchronous reads to ensure that the buffer doesn't get full. To avoid any deadlocks and collect up all output from both StandardOutput and StandardError you can do this:
EDIT: See answers below for how avoid an ObjectDisposedException if the timeout occurs.
using (Process process = new Process())
{
process.StartInfo.FileName = filename;
process.StartInfo.Arguments = arguments;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
StringBuilder output = new StringBuilder();
StringBuilder error = new StringBuilder();
using (AutoResetEvent outputWaitHandle = new AutoResetEvent(false))
using (AutoResetEvent errorWaitHandle = new AutoResetEvent(false))
{
process.OutputDataReceived += (sender, e) => {
if (e.Data == null)
{
outputWaitHandle.Set();
}
else
{
output.AppendLine(e.Data);
}
};
process.ErrorDataReceived += (sender, e) =>
{
if (e.Data == null)
{
errorWaitHandle.Set();
}
else
{
error.AppendLine(e.Data);
}
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
if (process.WaitForExit(timeout) &&
outputWaitHandle.WaitOne(timeout) &&
errorWaitHandle.WaitOne(timeout))
{
// Process completed. Check process.ExitCode here.
}
else
{
// Timed out.
}
}
}
The documentation for Process.StandardOutput says to read before you wait otherwise you can deadlock, snippet copied below:
// Start the child process.
Process p = new Process();
// Redirect the output stream of the child process.
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.FileName = "Write500Lines.exe";
p.Start();
// Do not wait for the child process to exit before
// reading to the end of its redirected stream.
// p.WaitForExit();
// Read the output stream first and then wait.
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
This is a more modern awaitable, Task Parallel Library (TPL) based solution for .NET 4.5 and above.
Usage Example
try
{
var exitCode = await StartProcess(
"dotnet",
"--version",
#"C:\",
10000,
Console.Out,
Console.Out);
Console.WriteLine($"Process Exited with Exit Code {exitCode}!");
}
catch (TaskCanceledException)
{
Console.WriteLine("Process Timed Out!");
}
Implementation
public static async Task<int> StartProcess(
string filename,
string arguments,
string workingDirectory= null,
int? timeout = null,
TextWriter outputTextWriter = null,
TextWriter errorTextWriter = null)
{
using (var process = new Process()
{
StartInfo = new ProcessStartInfo()
{
CreateNoWindow = true,
Arguments = arguments,
FileName = filename,
RedirectStandardOutput = outputTextWriter != null,
RedirectStandardError = errorTextWriter != null,
UseShellExecute = false,
WorkingDirectory = workingDirectory
}
})
{
var cancellationTokenSource = timeout.HasValue ?
new CancellationTokenSource(timeout.Value) :
new CancellationTokenSource();
process.Start();
var tasks = new List<Task>(3) { process.WaitForExitAsync(cancellationTokenSource.Token) };
if (outputTextWriter != null)
{
tasks.Add(ReadAsync(
x =>
{
process.OutputDataReceived += x;
process.BeginOutputReadLine();
},
x => process.OutputDataReceived -= x,
outputTextWriter,
cancellationTokenSource.Token));
}
if (errorTextWriter != null)
{
tasks.Add(ReadAsync(
x =>
{
process.ErrorDataReceived += x;
process.BeginErrorReadLine();
},
x => process.ErrorDataReceived -= x,
errorTextWriter,
cancellationTokenSource.Token));
}
await Task.WhenAll(tasks);
return process.ExitCode;
}
}
/// <summary>
/// Waits asynchronously for the process to exit.
/// </summary>
/// <param name="process">The process to wait for cancellation.</param>
/// <param name="cancellationToken">A cancellation token. If invoked, the task will return
/// immediately as cancelled.</param>
/// <returns>A Task representing waiting for the process to end.</returns>
public static Task WaitForExitAsync(
this Process process,
CancellationToken cancellationToken = default(CancellationToken))
{
process.EnableRaisingEvents = true;
var taskCompletionSource = new TaskCompletionSource<object>();
EventHandler handler = null;
handler = (sender, args) =>
{
process.Exited -= handler;
taskCompletionSource.TrySetResult(null);
};
process.Exited += handler;
if (cancellationToken != default(CancellationToken))
{
cancellationToken.Register(
() =>
{
process.Exited -= handler;
taskCompletionSource.TrySetCanceled();
});
}
return taskCompletionSource.Task;
}
/// <summary>
/// Reads the data from the specified data recieved event and writes it to the
/// <paramref name="textWriter"/>.
/// </summary>
/// <param name="addHandler">Adds the event handler.</param>
/// <param name="removeHandler">Removes the event handler.</param>
/// <param name="textWriter">The text writer.</param>
/// <param name="cancellationToken">The cancellation token.</param>
/// <returns>A task representing the asynchronous operation.</returns>
public static Task ReadAsync(
this Action<DataReceivedEventHandler> addHandler,
Action<DataReceivedEventHandler> removeHandler,
TextWriter textWriter,
CancellationToken cancellationToken = default(CancellationToken))
{
var taskCompletionSource = new TaskCompletionSource<object>();
DataReceivedEventHandler handler = null;
handler = new DataReceivedEventHandler(
(sender, e) =>
{
if (e.Data == null)
{
removeHandler(handler);
taskCompletionSource.TrySetResult(null);
}
else
{
textWriter.WriteLine(e.Data);
}
});
addHandler(handler);
if (cancellationToken != default(CancellationToken))
{
cancellationToken.Register(
() =>
{
removeHandler(handler);
taskCompletionSource.TrySetCanceled();
});
}
return taskCompletionSource.Task;
}
Mark Byers' answer is excellent, but I would just add the following:
The OutputDataReceived and ErrorDataReceived delegates need to be removed before the outputWaitHandle and errorWaitHandle get disposed. If the process continues to output data after the timeout has been exceeded and then terminates, the outputWaitHandle and errorWaitHandle variables will be accessed after being disposed.
(FYI I had to add this caveat as an answer as I couldn't comment on his post.)
The problem with unhandled ObjectDisposedException happens when the process is timed out. In such case the other parts of the condition:
if (process.WaitForExit(timeout)
&& outputWaitHandle.WaitOne(timeout)
&& errorWaitHandle.WaitOne(timeout))
are not executed. I resolved this problem in a following way:
using (AutoResetEvent outputWaitHandle = new AutoResetEvent(false))
using (AutoResetEvent errorWaitHandle = new AutoResetEvent(false))
{
using (Process process = new Process())
{
// preparing ProcessStartInfo
try
{
process.OutputDataReceived += (sender, e) =>
{
if (e.Data == null)
{
outputWaitHandle.Set();
}
else
{
outputBuilder.AppendLine(e.Data);
}
};
process.ErrorDataReceived += (sender, e) =>
{
if (e.Data == null)
{
errorWaitHandle.Set();
}
else
{
errorBuilder.AppendLine(e.Data);
}
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
if (process.WaitForExit(timeout))
{
exitCode = process.ExitCode;
}
else
{
// timed out
}
output = outputBuilder.ToString();
}
finally
{
outputWaitHandle.WaitOne(timeout);
errorWaitHandle.WaitOne(timeout);
}
}
}
Rob answered it and saved me few more hours of trials. Read the output/error buffer before waiting:
// Read the output stream first and then wait.
string output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
We have this issue as well (or a variant).
Try the following:
1) Add a timeout to p.WaitForExit(nnnn); where nnnn is in milliseconds.
2) Put the ReadToEnd call before the WaitForExit call. This is what we've seen MS recommend.
Credit to EM0 for https://stackoverflow.com/a/17600012/4151626
The other solutions (including EM0's) still deadlocked for my application, due to internal timeouts and the use of both StandardOutput and StandardError by the spawned application. Here is what worked for me:
Process p = new Process()
{
StartInfo = new ProcessStartInfo()
{
FileName = exe,
Arguments = args,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true
}
};
p.Start();
string cv_error = null;
Thread et = new Thread(() => { cv_error = p.StandardError.ReadToEnd(); });
et.Start();
string cv_out = null;
Thread ot = new Thread(() => { cv_out = p.StandardOutput.ReadToEnd(); });
ot.Start();
p.WaitForExit();
ot.Join();
et.Join();
Edit: added initialization of StartInfo to code sample
I solved it this way:
Process proc = new Process();
proc.StartInfo.FileName = batchFile;
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.RedirectStandardError = true;
proc.StartInfo.RedirectStandardInput = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
proc.Start();
StreamWriter streamWriter = proc.StandardInput;
StreamReader outputReader = proc.StandardOutput;
StreamReader errorReader = proc.StandardError;
while (!outputReader.EndOfStream)
{
string text = outputReader.ReadLine();
streamWriter.WriteLine(text);
}
while (!errorReader.EndOfStream)
{
string text = errorReader.ReadLine();
streamWriter.WriteLine(text);
}
streamWriter.Close();
proc.WaitForExit();
I redirected both input, output and error and handled reading from output and error streams.
This solution works for SDK 7- 8.1, both for Windows 7 and Windows 8
I tried to make a class that would solve your problem using asynchronous stream read, by taking in account Mark Byers, Rob, stevejay answers. Doing so I realised that there is a bug related to asynchronous process output stream read.
I reported that bug at Microsoft: https://connect.microsoft.com/VisualStudio/feedback/details/3119134
Summary:
You can't do that:
process.BeginOutputReadLine(); process.Start();
You will receive System.InvalidOperationException : StandardOut has
not been redirected or the process hasn't started yet.
============================================================================================================================
Then you have to start asynchronous output read after the process is
started:
process.Start(); process.BeginOutputReadLine();
Doing so, make a race condition because the output stream can receive
data before you set it to asynchronous:
process.Start();
// Here the operating system could give the cpu to another thread.
// For example, the newly created thread (Process) and it could start writing to the output
// immediately before next line would execute.
// That create a race condition.
process.BeginOutputReadLine();
============================================================================================================================
Then some people could say that you just have to read the stream
before you set it to asynchronous. But the same problem occurs. There
will be a race condition between the synchronous read and set the
stream into asynchronous mode.
============================================================================================================================
There is no way to acheive safe asynchronous read of an output stream
of a process in the actual way "Process" and "ProcessStartInfo" has
been designed.
You are probably better using asynchronous read like suggested by other users for your case. But you should be aware that you could miss some information due to race condition.
I think with async, it is possible to have a more elegant solution and not having deadlocks even when using both standardOutput and standardError:
using (Process process = new Process())
{
process.StartInfo.FileName = filename;
process.StartInfo.Arguments = arguments;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.Start();
var tStandardOutput = process.StandardOutput.ReadToEndAsync();
var tStandardError = process.StandardError.ReadToEndAsync();
if (process.WaitForExit(timeout))
{
string output = await tStandardOutput;
string errors = await tStandardError;
// Process completed. Check process.ExitCode here.
}
else
{
// Timed out.
}
}
It is base on Mark Byers answer.
If you are not in an async method, you can use string output = tStandardOutput.result; instead of await
I've read many of the answers and made my own. Not sure this one will fix in any case, but it fixes in my environment. I'm just not using WaitForExit and use WaitHandle.WaitAll on both output & error end signals. I will be glad, if someone will see possible problems with that. Or if it will help someone. For me it's better because not uses timeouts.
private static int DoProcess(string workingDir, string fileName, string arguments)
{
int exitCode;
using (var process = new Process
{
StartInfo =
{
WorkingDirectory = workingDir,
WindowStyle = ProcessWindowStyle.Hidden,
CreateNoWindow = true,
UseShellExecute = false,
FileName = fileName,
Arguments = arguments,
RedirectStandardError = true,
RedirectStandardOutput = true
},
EnableRaisingEvents = true
})
{
using (var outputWaitHandle = new AutoResetEvent(false))
using (var errorWaitHandle = new AutoResetEvent(false))
{
process.OutputDataReceived += (sender, args) =>
{
// ReSharper disable once AccessToDisposedClosure
if (args.Data != null) Debug.Log(args.Data);
else outputWaitHandle.Set();
};
process.ErrorDataReceived += (sender, args) =>
{
// ReSharper disable once AccessToDisposedClosure
if (args.Data != null) Debug.LogError(args.Data);
else errorWaitHandle.Set();
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
WaitHandle.WaitAll(new WaitHandle[] { outputWaitHandle, errorWaitHandle });
exitCode = process.ExitCode;
}
}
return exitCode;
}
I thing that this is simple and better approach (we don't need AutoResetEvent):
public static string GGSCIShell(string Path, string Command)
{
using (Process process = new Process())
{
process.StartInfo.WorkingDirectory = Path;
process.StartInfo.FileName = Path + #"\ggsci.exe";
process.StartInfo.CreateNoWindow = true;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.UseShellExecute = false;
StringBuilder output = new StringBuilder();
process.OutputDataReceived += (sender, e) =>
{
if (e.Data != null)
{
output.AppendLine(e.Data);
}
};
process.Start();
process.StandardInput.WriteLine(Command);
process.BeginOutputReadLine();
int timeoutParts = 10;
int timeoutPart = (int)TIMEOUT / timeoutParts;
do
{
Thread.Sleep(500);//sometimes halv scond is enough to empty output buff (therefore "exit" will be accepted without "timeoutPart" waiting)
process.StandardInput.WriteLine("exit");
timeoutParts--;
}
while (!process.WaitForExit(timeoutPart) && timeoutParts > 0);
if (timeoutParts <= 0)
{
output.AppendLine("------ GGSCIShell TIMEOUT: " + TIMEOUT + "ms ------");
}
string result = output.ToString();
return result;
}
}
None of the answers above is doing the job.
Rob solution hangs and 'Mark Byers' solution get the disposed exception.(I tried the "solutions" of the other answers).
So I decided to suggest another solution:
public void GetProcessOutputWithTimeout(Process process, int timeoutSec, CancellationToken token, out string output, out int exitCode)
{
string outputLocal = ""; int localExitCode = -1;
var task = System.Threading.Tasks.Task.Factory.StartNew(() =>
{
outputLocal = process.StandardOutput.ReadToEnd();
process.WaitForExit();
localExitCode = process.ExitCode;
}, token);
if (task.Wait(timeoutSec, token))
{
output = outputLocal;
exitCode = localExitCode;
}
else
{
exitCode = -1;
output = "";
}
}
using (var process = new Process())
{
process.StartInfo = ...;
process.Start();
string outputUnicode; int exitCode;
GetProcessOutputWithTimeout(process, PROCESS_TIMEOUT, out outputUnicode, out exitCode);
}
This code debugged and works perfectly.
Introduction
Currently accepted answer doesn't work (throws exception) and there are too many workarounds but no complete code. This is obviously wasting lots of people's time because this is a popular question.
Combining Mark Byers' answer and Karol Tyl's answer I wrote full code based on how I want to use the Process.Start method.
Usage
I have used it to create progress dialog around git commands. This is how I've used it:
private bool Run(string fullCommand)
{
Error = "";
int timeout = 5000;
var result = ProcessNoBS.Start(
filename: #"C:\Program Files\Git\cmd\git.exe",
arguments: fullCommand,
timeoutInMs: timeout,
workingDir: #"C:\test");
if (result.hasTimedOut)
{
Error = String.Format("Timeout ({0} sec)", timeout/1000);
return false;
}
if (result.ExitCode != 0)
{
Error = (String.IsNullOrWhiteSpace(result.stderr))
? result.stdout : result.stderr;
return false;
}
return true;
}
In theory you can also combine stdout and stderr, but I haven't tested that.
Code
public struct ProcessResult
{
public string stdout;
public string stderr;
public bool hasTimedOut;
private int? exitCode;
public ProcessResult(bool hasTimedOut = true)
{
this.hasTimedOut = hasTimedOut;
stdout = null;
stderr = null;
exitCode = null;
}
public int ExitCode
{
get
{
if (hasTimedOut)
throw new InvalidOperationException(
"There was no exit code - process has timed out.");
return (int)exitCode;
}
set
{
exitCode = value;
}
}
}
public class ProcessNoBS
{
public static ProcessResult Start(string filename, string arguments,
string workingDir = null, int timeoutInMs = 5000,
bool combineStdoutAndStderr = false)
{
using (AutoResetEvent outputWaitHandle = new AutoResetEvent(false))
using (AutoResetEvent errorWaitHandle = new AutoResetEvent(false))
{
using (var process = new Process())
{
var info = new ProcessStartInfo();
info.CreateNoWindow = true;
info.FileName = filename;
info.Arguments = arguments;
info.UseShellExecute = false;
info.RedirectStandardOutput = true;
info.RedirectStandardError = true;
if (workingDir != null)
info.WorkingDirectory = workingDir;
process.StartInfo = info;
StringBuilder stdout = new StringBuilder();
StringBuilder stderr = combineStdoutAndStderr
? stdout : new StringBuilder();
var result = new ProcessResult();
try
{
process.OutputDataReceived += (sender, e) =>
{
if (e.Data == null)
outputWaitHandle.Set();
else
stdout.AppendLine(e.Data);
};
process.ErrorDataReceived += (sender, e) =>
{
if (e.Data == null)
errorWaitHandle.Set();
else
stderr.AppendLine(e.Data);
};
process.Start();
process.BeginOutputReadLine();
process.BeginErrorReadLine();
if (process.WaitForExit(timeoutInMs))
result.ExitCode = process.ExitCode;
// else process has timed out
// but that's already default ProcessResult
result.stdout = stdout.ToString();
if (combineStdoutAndStderr)
result.stderr = null;
else
result.stderr = stderr.ToString();
return result;
}
finally
{
outputWaitHandle.WaitOne(timeoutInMs);
errorWaitHandle.WaitOne(timeoutInMs);
}
}
}
}
}
I know that this is supper old but, after reading this whole page none of the solutions was working for me, although I didn't try Muhammad Rehan as the code was a little hard to follow, although I guess he was on the right track. When I say it didn't work that's not entirely true, sometimes it would work fine, I guess it is something to do with the length of the output before an EOF mark.
Anyway, the solution that worked for me was to use different threads to read the StandardOutput and StandardError and write the messages.
StreamWriter sw = null;
var queue = new ConcurrentQueue<string>();
var flushTask = new System.Timers.Timer(50);
flushTask.Elapsed += (s, e) =>
{
while (!queue.IsEmpty)
{
string line = null;
if (queue.TryDequeue(out line))
sw.WriteLine(line);
}
sw.FlushAsync();
};
flushTask.Start();
using (var process = new Process())
{
try
{
process.StartInfo.FileName = #"...";
process.StartInfo.Arguments = $"...";
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.Start();
var outputRead = Task.Run(() =>
{
while (!process.StandardOutput.EndOfStream)
{
queue.Enqueue(process.StandardOutput.ReadLine());
}
});
var errorRead = Task.Run(() =>
{
while (!process.StandardError.EndOfStream)
{
queue.Enqueue(process.StandardError.ReadLine());
}
});
var timeout = new TimeSpan(hours: 0, minutes: 10, seconds: 0);
if (Task.WaitAll(new[] { outputRead, errorRead }, timeout) &&
process.WaitForExit((int)timeout.TotalMilliseconds))
{
if (process.ExitCode != 0)
{
throw new Exception($"Failed run... blah blah");
}
}
else
{
throw new Exception($"process timed out after waiting {timeout}");
}
}
catch (Exception e)
{
throw new Exception($"Failed to succesfully run the process.....", e);
}
}
}
Hope this helps someone, who thought this could be so hard!
After reading all the posts here, i settled on the consolidated solution of Marko Avlijaš.
However, it did not solve all of my issues.
In our environment we have a Windows Service which is scheduled to run hundreds of different .bat .cmd .exe,... etc. files which have accumulated over the years and were written by many different people and in different styles. We have no control over the writing of the programs & scripts, we are just responsible for scheduling, running, and reporting on success/failure.
So i tried pretty much all of the suggestions here with different levels of success. Marko's answer was almost perfect, but when run as a service, it didnt always capture stdout. I never got to the bottom of why not.
The only solution we found that works in ALL our cases is this : http://csharptest.net/319/using-the-processrunner-class/index.html
Workaround I ended up using to avoid all the complexity:
var outputFile = Path.GetTempFileName();
info = new System.Diagnostics.ProcessStartInfo("TheProgram.exe", String.Join(" ", args) + " > " + outputFile + " 2>&1");
info.CreateNoWindow = true;
info.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
info.UseShellExecute = false;
System.Diagnostics.Process p = System.Diagnostics.Process.Start(info);
p.WaitForExit();
Console.WriteLine(File.ReadAllText(outputFile)); //need the StandardOutput contents
So I create a temp file, redirect both the output and error to it by using > outputfile > 2>&1 and then just read the file after the process has finished.
The other solutions are fine for scenarios where you want to do other stuff with the output, but for simple stuff this avoids a lot of complexity.
In my case I had an error so I just waited in vain for a normal ouput.
I switched the order from this:
string result = process.StandardOutput.ReadToEnd();
string error = process.StandardError.ReadToEnd();
To this:
string error = process.StandardError.ReadToEnd();
if (string.IsNullOrEmpty(error))
string result = process.StandardOutput.ReadToEnd();
This post maybe outdated but i found out the main cause why it usually hang is due to stack overflow for the redirectStandardoutput or if you have redirectStandarderror.
As the output data or the error data is large, it will cause a hang time as it is still processing for indefinite duration.
so to resolve this issue:
p.StartInfo.RedirectStandardoutput = False
p.StartInfo.RedirectStandarderror = False
Let us call the sample code posted here the redirector and the other program the redirected. If it were me then I would probably write a test redirected program that can be used to duplicate the problem.
So I did. For test data I used the ECMA-334 C# Language Specificationv PDF; it is about 5MB. The following is the important part of that.
StreamReader stream = null;
try { stream = new StreamReader(Path); }
catch (Exception ex)
{
Console.Error.WriteLine("Input open error: " + ex.Message);
return;
}
Console.SetIn(stream);
int datasize = 0;
try
{
string record = Console.ReadLine();
while (record != null)
{
datasize += record.Length + 2;
record = Console.ReadLine();
Console.WriteLine(record);
}
}
catch (Exception ex)
{
Console.Error.WriteLine($"Error: {ex.Message}");
return;
}
The datasize value does not match the actual file size but that does not matter. It is not clear if a PDF file always uses both CR and LF at the end of lines but that does not matter for this. You can use any other large text file to test with.
Using that the sample redirector code hangs when I write the large amount of data but not when I write a small amount.
I tried very much to somehow trace the execution of that code and I could not. I commented out the lines of the redirected program that disabled creation of a console for the redirected program to try to get a separate console window but I could not.
Then I found How to start a console app in a new window, the parent’s window, or no window. So apparently we cannot (easily) have a separate console when one console program starts another console program without ShellExecute and since ShellExecute does not support redirection we must share a console, even if we specify no window for the other process.
I assume that if the redirected program fills up a buffer somewhere then it must wait for the data to be read and if at that point no data is read by the redirector then it is a deadlock.
The solution is to not use ReadToEnd and to read the data while the data is being written but it is not necessary to use asynchronous reads. The solution can be quite simple. The following works for me with the 5 MB PDF.
ProcessStartInfo info = new ProcessStartInfo(TheProgram);
info.CreateNoWindow = true;
info.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
info.RedirectStandardOutput = true;
info.UseShellExecute = false;
Process p = Process.Start(info);
string record = p.StandardOutput.ReadLine();
while (record != null)
{
Console.WriteLine(record);
record = p.StandardOutput.ReadLine();
}
p.WaitForExit();
Another possibility is to use a GUI program to do the redirection. The preceding code works in a WPF application except with obvious modifications.
I was having the same issue, but the reason was different. It would however happen under Windows 8, but not under Windows 7. The following line seems to have caused the problem.
pProcess.StartInfo.UseShellExecute = False
The solution was to NOT disable UseShellExecute. I now received a Shell popup window, which is unwanted, but much better than the program waiting for nothing particular to happen. So I added the following work-around for that:
pProcess.StartInfo.WindowStyle = ProcessWindowStyle.Hidden
Now the only thing bothering me is to why this is happening under Windows 8 in the first place.
hi i'm trying to build a parser for my System to Manage my Tekkit Server i am using C# but i have RedirectStandardOutput on my Tekkit Server process and there is a method set-up to then send that output to my console after adding to a List but it's not adding to a List<string>
Here is my code:
public void StartServer(string maxMem, string minMem, string path)
{
ThreadStart server = new ThreadStart(delegate() { StartServerThread(maxMem, minMem, path); });
server.Invoke();
}
private void StartServerThread(string maxMem, string minMem, string TekkitPath)
{
try
{
TekkitServer.StartInfo.FileName = "java";
TekkitServer.StartInfo.Arguments = String.Format("-Xmx{0} -Xms{1} -jar \"" + TekkitPath + "\" -nojline nogui", maxMem, minMem);
TekkitServer.StartInfo.UseShellExecute = false;
TekkitServer.StartInfo.RedirectStandardInput = true;
TekkitServer.StartInfo.RedirectStandardOutput = true;
TekkitServer.OutputDataReceived += new DataReceivedEventHandler(TekkitServer_OutputDataReceived);
IsStarted = TekkitServer.Start();
TekkitServerInput = TekkitServer.StandardInput;
}
catch (Exception)
{
}
}
void TekkitServer_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
/*B*/recordedData.Add(e.Data);
Console.Out.WriteLine(e.Data);
}
Where /*B*/ is a break point, the breakpoint is never activating
By default, the standard output is directed at the console window.
If you need to do something with it, you need to redirect it, hence, you need to set RedirectStandardOutput = true; for the event to be fired.
Edit: This is my working code (with error handling and logging omitted):
public int ExecuteCommand(CommandParameters parameters)
{
Process process = new Process();
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.RedirectStandardError = true;
process.OutputDataReceived += StdOutputHandler;
process.ErrorDataReceived += StdErrorHandler;
process.StartInfo.UseShellExecute = false;
process.StartInfo.FileName = ...;
process.StartInfo.Arguments = ...;
process.Start();
process.BeginErrorReadLine();
process.BeginOutputReadLine();
process.WaitForExit(parameters.Timeout);
return process.ExitCode;
}
private void StdOutputHandler(object sendingProcess, DataReceivedEventArgs outdata)
{
if (!string.IsNullOrEmpty(outdata.Data))
{
OutputMessages.Add(outdata.Data);
}
}
Most likely the missing link in your code is the BeginOutputReadLine that actually gets the handler method on it's way.
Also, I use a fresh Process object and that I wait on it to finish it's job, so no interference with previous calls is possible.