We Keep Coding

What is breaking my Process.StartInfo.OutputDataReceived callbacks prematurely?

The following problem occurs on .NET Framework v3.5. Don't know if it applies to v4*.
To capture stdout for some processes I've successfully used p.StartInfo.UseShellExecute = false; and p.StartInfo.RedirectStandardOutput = true; and an event handler hooked to p.StartInfo.OutputDataReceived+=...;. Then I call p.Start(); then p.BeginOutputReadLine(); and then p.WaitForExit();
All is well so far. I get all stdout on the event handler, line by line, as expected.
I had to introduce a timeout instead of WaitForExit() because some processes unpredictably trigger requests for input at stdin (e.g. are you sure? [y/n]) leading to a deadlock where I wait forever and so do they.
The first thing I tried is changing to while (!p.HasExited && DateTime.Now < timeoutmom) p.WaitForExit(200); where timeoutmoment is 2 minutes after proc.Start(). This is when I ran into problems. Very consistently, the code works for calls that produce up to a few hundred lines of stdout but it breaks for one call that produces about 7500 lines. What happens is the proc.WaitForExit(200); thread exits the while when my OutputDataReceived event handler was called for only ~ 7300 lines (this number is again very consistent it varies by only +/- 1 between tests) and the handler is not called anymore for the rest of the stdout lines so I lose them.
Strangely, the problem doesn't appear if I avoid WaitForExit(200) and instead use while (!p.HasExited && DateTime.Now < timeoutmom) System.Threading.Thread.Sleep(1000); (not shown in the code below). When I posted the question I was pretty sure the problem was avoided using Sleep(1000) but I was wrong. It worked a few dozen times like that and then it didn't, it started behaving just like when I checked WaitForExit(200).
I now speculate that the reasons for this problem are (1) I take too long to process each OutputDataReceived callback. I noticed the problem was aggravated when I added a conditional breakpoint in the event handler which lengthened the method execution by a lot. I can now reproduce the problem by simply adding 3x Debug.WriteLines without the conditional breakpoint; PLUS (2) my context is somehow corrupted by me accessing HasExited / WaitForExit(200) before the system had a chance to perform all the callbacks on my event handler. I now do a blind System.Threading.Thread.Sleep(30000) just after p.Start() and before accessing any p.* method and I get all the callbacks. When I used WaitForExit() it seemed I can take however much time I want to process every callback and I would still get them all.
Can someone make more sense of this?
Code:
private int _execOsProc(
ProcessStartInfo Psi
, string SecInsensArgs
, TextWriter ExtraStdOutAndErrTgt
, bool OutputToExtraStdOutOnly
)
{
var pr = new Process();
pr.StartInfo = Psi;
pr.StartInfo.UseShellExecute = false;
pr.StartInfo.RedirectStandardOutput = pr.StartInfo.RedirectStandardError = true;
pr.StartInfo.CreateNoWindow = true;
var ol = new DataReceivedEventHandler(this._stdOutDataReceived);
var el = new DataReceivedEventHandler(this._stdErrDataReceived);
pr.OutputDataReceived += ol;
pr.ErrorDataReceived += el;
try
{
__logger.Debug("Executing: \"" + pr.StartInfo.FileName + "\" " + SecInsensArgs);
if (ExtraStdOutAndErrTgt == null)
{
this.__outputToExtraStdOutOnly = false;
}
else
{
this.__extraStdOutAndErrTgt = ExtraStdOutAndErrTgt;
this.__outputToExtraStdOutOnly = OutputToExtraStdOutOnly;
}
pr.Start();
pr.BeginOutputReadLine();
pr.BeginErrorReadLine();
var startmom = DateTime.Now;
var timeoutmom = startmom.AddMinutes(2);
while (!pr.HasExited && DateTime.Now < timeoutmom) pr.WaitForExit(200);
pr.CancelOutputRead();
pr.CancelErrorRead();
if (pr.HasExited)
{
__logger.Debug("Execution finished with exit status code: " + pr.ExitCode);
return pr.ExitCode;
}
else
{
__logger.Debug("Timeout while waiting for execution to finish");
pr.Kill();
return -100;
}
}
finally
{
pr.OutputDataReceived -= ol;
pr.ErrorDataReceived -= el;
if (this.__extraStdOutAndErrTgt != null)
{
this.__extraStdOutAndErrTgt = null;
this.__outputToExtraStdOutOnly = false;
}
}
}
private void _stdOutDataReceived(
object sender
, DataReceivedEventArgs e
)
{
string rdata = string.IsNullOrEmpty(e.Data) ? "" : e.Data.Trim();
if (!this.__outputToExtraStdOutOnly) __logger.Debug("SO: " + rdata);
if (this.__extraStdOutAndErrTgt != null)
{
lock (this.__extraStdOutAndErrTgt)
{
try
{
this.__extraStdOutAndErrTgt.WriteLine(rdata);
this.__extraStdOutAndErrTgt.Flush();
}
catch (Exception exc)
{
__logger.Warn(
"WARNING: Error detected but ignored during extra stream write"
+ " on SODR. Details: " + exc.Message
, exc
);
}
}
}
}
private void _stdErrDataReceived(
object sender
, DataReceivedEventArgs e
)
{
string rdata = string.IsNullOrEmpty(e.Data) ? "" : e.Data.Trim();
if (!__outputToExtraStdOutOnly) __logger.Debug("SE: " + rdata);
if (this.__extraStdOutAndErrTgt != null)
{
lock (this.__extraStdOutAndErrTgt)
{
try
{
this.__extraStdOutAndErrTgt.WriteLine(rdata);
this.__extraStdOutAndErrTgt.Flush();
}
catch (Exception exc)
{
__logger.Warn(
"WARNING: Error detected but ignored during extra stream write"
+ " on SEDR. Details: " + exc.Message
, exc
);
}
}
}
}

I'm not sure if it will solve the problem, but it is too long to post it in the comment.
MSDN says about Process.HasExited:
When standard output has been redirected to asynchronous event
handlers, it is possible that output processing will not have
completed when this property returns true. To ensure that asynchronous
event handling has been completed, call the WaitForExit() overload
that takes no parameter before checking HasExited.
and about WaitForExit():
This overload ensures that all processing has been completed,
including the handling of asynchronous events for redirected standard
output. You should use this overload after a call to the
WaitForExit(Int32) overload when standard output has been redirected
to asynchronous event handlers.
It indicates, that call to WaitForExit() with no parameters should solve the problem. Something like:
var startmom = DateTime.Now;
var timeoutmom = startmom.AddMinutes(2);
while (!pr.HasExited && DateTime.Now < timeoutmom)
pr.WaitForExit(200);
if (pr.HasExited)
{
WaitForExit();//Ensure that redirected output buffers are flushed
pr.CancelOutputRead();
pr.CancelErrorRead();
__logger.Debug("Execution finished with exit status code: " + pr.ExitCode);
return pr.ExitCode;
}
else
{
pr.CancelOutputRead();
pr.CancelErrorRead();
__logger.Debug("Timeout while waiting for execution to finish");
pr.Kill();
return -100;
}

C# Timeout with Solidworks VBA Macro

I have a few functions in a Solidworks Addin which call on a VBA macro (Via the runMacro2 method) a co-worker has been working on for the last few weeks. In his code he calls a Solidworks function which, under certain, unknown conditions, hangs for a long period of time. How long seems to depend upon the size and quantity of bodies in the part. Considering at least one of the functions we want to run this from i automatic, this just wont do.
I have tried using the Thread.Join(int) method (shown below) but it doesnt work. I also tried modifying the code from this answer Close a MessageBox after several seconds with the same results. Is there anything I can do either in C# or VBA to handle a timeout for this without re-writing his entire macro?
public void runBB()
{
Stopwatch testStop = new Stopwatch();
Thread workerThread = new Thread(bbRun);
testStop.Start();
workerThread.Start();
if (!workerThread.Join(50))
{
workerThread.Abort();
testStop.Stop();
MessageBox.Show("Unable to generate Bounding Box after " + testStop.ElapsedMilliseconds/1000 + " seconds. Please enter data manually.", "Solidworks Derped Error.");
}
return;
}//Still uses Macro (2-5-16)
public static void bbRun()
{
iSwApp.RunMacro2(macroPath + "BOUNDING_BOX.swp", "test11", "main", 0, out runMacroError);
return;
}

I was getting this same exact issue with SOLIDWORKS hanging on an open of a file. Almost all reference on SO was that you should never do this, but in this scenario, you either have to close it or wait forever. In C# I created a callWithTimeout method:
private void callWithTimeout(Action action, int timeoutMilliseconds, String errorText) {
Thread threadToKill = null;
Action wrappedAction = () =>
{
threadToKill = Thread.CurrentThread;
action();
};
IAsyncResult result = wrappedAction.BeginInvoke(null, null);
if (result.AsyncWaitHandle.WaitOne(timeoutMilliseconds)) {
wrappedAction.EndInvoke(result);
} else {
threadToKill.Abort();
throw new TimeoutException(errorText);
}
}
Then the code that was hanging put in a block as such:
bool timedOut = false;
try {
callWithTimeout(delegate() {
// code that hangs here
}, 60000, "Operation timed out. SOLIDWORKS could not open the file. This file will be processed later.");
} catch (TimeoutException){
timedOut = true;
} finally {
if(timedOut) {
Process[] prs = Process.GetProcesses();
foreach (Process p in prs) {
if (p?.ProcessName.Equals("SLDWORKS") ?? false)
p?.Kill();
}
}
}

Why sometimes when watching a file/directory for changes it's not changing the flag to true?

In the constructor i'm calling WatchDirectory method:
private void WatchDirectory()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = userVideosDirectory;
watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size;
watcher.Filter = "*.mp4";
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.EnableRaisingEvents = true;
}
Then the event OnChanged:
private void OnChanged(object source, FileSystemEventArgs e)
{
try
{
var info = new FileInfo(e.FullPath);
fileforupload = info.FullName;
if (e.ChangeType == WatcherChangeTypes.Changed)
{
var theSize = info.Length;
label2.BeginInvoke((Action)(() =>
{
label2.Text = theSize.ToString();
}));
}
dirchanged = true;
}
catch (Exception ee)
{
string err = ee.ToString();
}
}
Then i'm using a while loop to check when dirchange flag is true:
WatchDirectory();
while (dirchanged == false)
{
if (dirchanged == true)
{
Youtube_Uploader youtubeupload = new
Youtube_Uploader(fileforupload);
break;
}
}
The problem is that sometimes it's never changes the dirchanged to true on the OnChanged event. Not sure why. It seems to fire the OnChanged event but sometimes it doesn't execute the dirchanged = true;
Therefore inside the while loop dirchanged flag remains false all the time.
I added now a new method i called it IsFileLocked:
protected virtual bool IsFileLocked(FileInfo file)
{
FileStream stream = null;
try
{
stream = file.Open(FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException)
{
return true;
}
finally
{
if (stream != null)
stream.Close();
}
return false;
}
And i use this in the event OnChanged:
private void OnChanged(object source, FileSystemEventArgs e)
{
try
{
var info = new FileInfo(e.FullPath);
fileforupload = info.FullName;
IsFileLocked(info);
if (e.ChangeType == WatcherChangeTypes.Changed)
{
var theSize = info.Length;
label2.BeginInvoke((Action)(() =>
{
label2.Text = theSize.ToString();
}));
}
dirchanged = true;
}
catch (Exception ee)
{
string err = ee.ToString();
}
}
And in the method IsFileLocked i'm getting exception:
The process cannot access the file 'C:\Users\bout0_000\Videos\My Great Game - My Great Capture - 2015-08-10 14-22-52.mp4' because it is being used by another process.
I'm using external program that create the file and since the program still working on creating the file the watcher can't get to it.
So i have a confilct here from one side i want to know to watch when the file is ready finished created but on the other side i can't know since the external program still working on it.
So how can i find out when the external program finished working on the file and the file is ready ?
This is the whole part of the code of the while:
if (request.QueryString[0] == "stop")
{
dirchanged = false;
StartRecrod();
result = "Recording stopped and preparing the file to be shared on youtube";
WatchDirectory();
while (dirchanged == false)
{
if (dirchanged == true)
{
string ttttt = "ok";
break;
}
}
}
I added a string ttttt just for testing.
Sometimes it's getting to the string ttttt when using a break point and sometimes not.
In my program when i touch my android screen it send command to the pc web server and it's getting here but someting is wrong with the while loop and the flag dirchanged sometimes it does enter the while and the IF and does the string ttttt and sometimes it dosen't.
This is what i did now with the await:
TaskCompletionSource<bool> sy;
public async void SendResponse(HttpListenerRequest request)
{
string result = "";
string key = request.QueryString.GetKey(0);
if (key == "cmd")
{
if (request.QueryString[0] == "nothing")
{
return "Connection Success";
}
if (request.QueryString[0] == "start")
{
StartRecrod();
result = "Recording started";
}
if (request.QueryString[0] == "stop")
{
dirchanged = false;
StartRecrod();
result = "Recording stopped and preparing the file to be shared on youtube";
sy = new TaskCompletionSource<bool>();
WatchDirectory();
await sy.Task;
Youtube_Uploader youtubeupload = new Youtube_Uploader(fileforupload);
}
}
else
{
result = "Nothing have been done";
}
if (Youtube_Uploader.fileuploadedsuccess != null && Youtube_Uploader.fileuploadedsuccess != "")
{
result = Youtube_Uploader.fileuploadedsuccess;
}
return result;
}
But some problems.
First i'm getting errors over all the returns.
Error 2 Since 'Automatic_Record.Form1.SendResponse(System.Net.HttpListenerRequest)' returns void, a return keyword must not be followed by an object expression
And error when init my web server:
WebServer ws = new WebServer(SendResponse, "http://+:8098/");
On the SendResponse i'm getting:
Error 1 'void Automatic_Record.Form1.SendResponse(System.Net.HttpListenerRequest)' has the wrong return type
This errors happen now when changed the method to async.
This is my WebServer method that i get error when init it since it should get something else then async:
public WebServer(Func<HttpListenerRequest, string> method, params string[] prefixes)
: this(prefixes, method) { }
public void Run()
{
ThreadPool.QueueUserWorkItem((o) =>
{
Console.WriteLine("Webserver running...");
try
{
while (_listener.IsListening)
{
ThreadPool.QueueUserWorkItem((c) =>
{
var ctx = c as HttpListenerContext;
try
{
string rstr = _responderMethod(ctx.Request);
System.Diagnostics.Trace.Write(ctx.Request.QueryString);
//ctx.Request.QueryString
byte[] buf = Encoding.UTF8.GetBytes(rstr);
ctx.Response.ContentLength64 = buf.Length;
ctx.Response.OutputStream.Write(buf, 0, buf.Length);
System.Data.SqlClient.SqlConnectionStringBuilder builder = new System.Data.SqlClient.SqlConnectionStringBuilder();
}
catch { } // suppress any exceptions
finally
{
// always close the stream
ctx.Response.OutputStream.Close();
}
}, _listener.GetContext());
}
}
catch { } // suppress any exceptions
});
}

This code is horribly broken. Yes, dirchanged is always false inside the while loop, because if it becomes true you won't be in the while loop any longer.
In addition, your code blocks events from occurring, which may block the file watcher event itself, and also is not optimization safe. Use proper synchronization, here's an example:
TaskCompletionSource<bool> sy;
private void OnChanged(object source, FileSystemEventArgs e)
{
sy.SetResult(true);
}
and wait with
sy = new TaskCompletionSource<bool>();
WatchDirectory();
await sy.Task; // or sy.Task.Wait()
(You'll need to use the async keyword on the method containing that code).
This fixes all the problems you had before -- it doesn't burn CPU cycles, it continues processing Windows messages, and it won't break if the compiler chooses to cache variables in registers.

dirchanged could be getting set to true just after evaluating the inner if block. Then, next loop it breaks out without ever running your uploader.

So you have two main questions?
1.) Why is dirchanged not being set to true?
and the apparent cause...
2.) How do you use FileSystemWatcher to only act on a file that's available for edit?
FileSystemWatcher is known for being a little touchy, and I agree with your diagnosis that file access is probably the culprit. An unpredictable file access error is exactly what I would expected from a FileSystemWatcher trying to do something with a file that was just modified. Can you edit the code that's creating the file? If so, one method I've used with FileSystemWatcher is to have it only watch for file creation of a fictitious file type such as ".fsw". The program creating the file will then rename it to ".fsw" whenever it is done editing it, that way the FileSystemWatcher only gets called when it has a file available to act upon, and it can then rename the file to it's actual type. Also, if you can edit the creation code, make sure that you are doing everything you can to release the file from there. I've seen this behavior before because I forgot to close a TextWriter.
Also, I would move the line
dirchanged = true;
Outside of the try statement. Why have it in there since it definitely wont throw an error? Also, your catch statement isn't really doing error handling. Any error in your try statement and you get ejected before reaching the dirchanged = true line without being alerted that this is what happened. Have you tested your delegate code on its own? Is there a need to have the if statement for type = changed right there? If you're troubleshooting, I would consider limiting your try statement content or moving it to after the while loop as much as possible.
Also, wouldn't this be a lot more simple for your while statement?
while (dirchanged == false){}
Youtube_Uploader youtubeupload = new Youtube_Uploader(fileforupload);
It's not the most elegant solution, but one work around is to simply wait if you know the program creating/editing the file is going to close it very soon...
while (dirchanged == false){}
System.Threading.Thread.Sleep(1000);
Youtube_Uploader youtubeupload = new Youtube_Uploader(fileforupload);
EDIT: Better yet, rather than a while statement use Ben Voigt's suggestion of a TaskCompletionSource. You'll still have to deal with the file being locked but you should be able to do that after the "task" has been flagged as completed.

How to end thread in win CF

Windows mobile 5; compact framework and relative newbie to c# and threads.
I want to download large files (several meg) from my own website; being GPRS this could take a while. I want to show a progress bar, and allow an option to cancel the download.
I've got a class called FileDownload and create an instance of it; give it a url and save location then:
MyFileDownLoader.Changed += new FileDownLoader.ChangedEventHandler(InvokeProgressBar);
BGDownload = new Thread(new ThreadStart(MyFileDownLoader.DownloadFile));
BGDownload.Start();
So I create an event handler for updates to progress bar and start the thread. This works fine.
I've got a cancel button which reads:
MyFileDownLoader.Changed -= InvokeProgressBar;
MyFileDownLoader.Cancel();
BGDownload.Join();
lblPercentage.Text = CurPercentage + " Cancelled"; // CurPercentage is a string
lblPercentage.Refresh();
btnUpdate.Enabled = true;
In the FileDownload class the key parts are:
public void Cancel()
{
CancelRequest = true;
}
In method Download file:
...
success = false;
try {
//loop until no data is returned
while ((bytesRead = responseStream.Read(buffer, 0, maxRead)) > 0)
{
_totalBytesRead += bytesRead;
BytesChanged(_totalBytesRead);
fileStream.Write(buffer, 0, bytesRead);
if (CancelRequest)
break;
}
if (!CancelRequest)
success = true;
}
catch
{
success = false;
// other error handling code
}
finally
{
if (null != responseStream)
responseStream.Close();
if (null != response)
response.Close();
if (null != fileStream)
fileStream.Close();
}
// if part of the file was written and the transfer failed, delete the partial file
if (!success && File.Exists(destination))
File.Delete(destination);
The code i'm using for the download is based on http://spitzkoff.com/craig/?p=24
The problem i've got is when I cancel, the download stops immediately, however it can take up to 5 seconds or so for the join process to complete. This is evidenced by lblPercentage.Text being updated after the join.
If I then try and download again, sometimes it works and sometimes I get a nullreference exception (still trying to track that down).
I think i'm doing something wrong in my approach to cancelling the thread.
Am i ?

public void Cancel()
{
CancelRequest = true;
}
I suppose you should add thread-safe to this action.
public void Cancel()
{
lock (this)
{
CancelRequest = true;
}
}
Hope this help!

Manipulate the output receiving from command RUN and act accordingly - C#

I guess you all misunderstood my question and closed it at How to run a Command in C# and retrieve data from it?
I had said in that post also :-
want to run a Command from command promt and want its output and maipulate its output. If required, want to close the process and display error or appropriate message. To stop the process, I have to press "F4' key on command prompt. Till the process is stopeed or killed, it has to be alive only.
I have created a class to handle running the cmd. And I keep getting the output. But on reading the output's each line I want to stop or throw exception if found anything improper in the output.
I am tying to connect to server via cmd. Server keeps on giving output. Suppose the server gave output as :
Trying to start .....
Cananot load file
.....
Exiting
While retrieving the output, I want to check for lines like "Cannot find file", "Connected Successfully, etc and set properties ccordingly (like connected = true, errorMsg = "Cannot find file". Where I am calling the class, I can take care of those proeprties and stop if found connected == true or errorMsg.length > 0. With this inform the user that "Connection is achieved or error msg stating regarding "Cannot load file" and disconnect the server if errorMsg found.
I didn't find anywhere doing any manipulation on the output receving and that's where I find myself stuck. I found a lot on internet. Am stuck and trying to figre out this part from last 3-4 days. Then have posted here.
I need help in that. Please help me. If requiried I will psot code snippets. But please help me. AND don't close this thread as answered ithout understanding my question fully. This is no duplicate.
My code is class :
public int ConnectToServer()
{
int error = 0;
connected = false;
try
{
process = Process.Start(processInfo);
process.BeginOutputReadLine();
process.OutputDataReceived += new DataReceivedEventHandler(Process_OutputDataReceived);
//if (errorMsg.Length > 0)
// throw new Exception(errorMsg);
}
catch (Exception e)
{
Console.WriteLine("Error Processing ConnectToServer : " + e.Message);
connected = false;
errorMsg = e.Message;
error = -1;
return error;
}
return error;
}
private void Process_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
errorMsg = "";
connected = false;
string d = e.Data;
if (!string.IsNullOrEmpty(d))
{
if (sb != null)
sb.Append(d + "\n");
Console.WriteLine("LINE = " + d);
if (d.IndexOf("Initialization Completed") > 0)
{
connected = true;
Console.WriteLine("********* Connected = " + connected);
}
else if (isInValidLine(d))
{
//throw new Exception(d);
connected = false;
errorMsg = d;
return;
}
}
return;
}
private bool isInValidLine(string line)
{
if (line.IndexOf("Cannot load file") > 0)
{
errorMsg = line;
return true;
}
return false;
}
IS THE ABOVE CLASS CODE CORRECT WITH MY REQUIREMENTS ?
In impementation :
while (!oc.Connected)
{
timepassed = (int)(DateTime.Now - start).TotalMilliseconds;
if (timepassed > timeout)
{
oc.DisconnectServer();
connectedToVpn = false;
throw new Exception("NotConnectedException");
} else if (oc.ErrorMessage.Length > 0)
{
oc.DisconnectServer();
connectedToVpn = false;
throw new Exception(oc.ErrorMessage);
}
Thread.Sleep(100);
}
Here what I am doing is, when I get the output line, I check if it states as Conneced or is invalid. If its invalid, I set the line as the errorMsg. In my while loop I keep chekcing for Connected and errorMessage, but the value of errorMessage stays as "" only. It never gets updated, which tell me that the processing output code is never executed. Nor in debug mode I find the cursor at that line, but the Line = is displayed proeprly in Console. So, don't understand what's going wrong and where.
Hope this helps you more understand.
Thanks

once you have redirected the standard output of the process you have executed you could parse what you receive as it arrives, I believe also line by line, then you can post commands to control your process.
to read output you have redirected the standard output, to send input you should also redirect the standard input of the process.