We Keep Coding

Re-synchronize Process.RedirectStandardOutput

Background
I'm writing a c# wrapper for a node.js application. In this wrapper I continuously read the standard output via Process.RedirectStandardOutput. The event is bound to the function onOutputDataReceived, in an instance of the class ProcessManager. In this same instance, there is also an instance of a custom event system.
[ProcessManager]
EventSystem eventSystem;
private void Start()
{
[...]
process.OutputDataReceived += onOutputDataReceived;
[...]
}
private void onOutputDataReceived(object sender, DataReceivedEventArgs e)
{
[...]
eventSystem.call(eventName, args);
}
[EventSystem]
List<EventHandler> eventList;
public Boolean call(String eventName, dynamic args)
{
[...]
foreach (EventHandler handler in eventList)
{
handler(args);
}
[...]
}
The problem occurs when the event is being called. Here is an example from a winforms application using my wrapper.
Wrapper.ProcessManager procMan;
procMan.eventSystem.on(eventName, (a) =>
{
button1.Text = someValue;
});
When run, the application crashes with the message
Cross-thread operation not valid: Control 'button1' accessed from a thread other than the thread it was created on
My issue, as I understand it, is this:
onOutputDataReceived is being executed asynchronously, in its own thread. As this same thread, only meant to be handling the output, goes on to call the events, I'm unintentionally multithreading my wrapper, making life harder for anyone implementing it.
Basically,
I need to run the line eventSystem.call() in the same thread that maintains the rest of the ProcessManager instance, as soon as new output data has been received as possible. Any ideas on how this best can be achieved?
A solution I've thought of is something like this
[ProcessManager]
Queue<string> waiting = new Queue<string();
EventSystem eventSystem;
private void onOutputDataReceived(object sender, DataReceivedEventArgs e)
{
[...]
waiting.Enqueue(eventName);
}
private void WhenReady()
{
while(waiting.Count > 0)
eventSystem.call(waiting.Dequeue());
}
As far as I can see, this would involve some kind of polling every x milliseconds, which doesn't feel like a clean solution. Also, it seems to me as if such a solution would be way too expensive for when no messages are being received and too slow for when some are.

The code that executes the nodejs process and reads its output should not need to know about the threading requirements of event subscribers. Make the subscriber satisfy its own requirements:
(a) =>
{
Invoke(new Action(() => button1.Text = someValue)); //marshal to UI thread
}
Your tentative solution would not work because it would block the UI thread.
Also, waiting is being used in an unsynchronized way... This is an unrelated bug.

C# Is it safe to call 'BackgroundWorker' inside 'using' statement?

I have developed a windows service. In the service I was using a BackgroundWorker to Post data in my Database.
I declared a BackgroundWorker inside my database constructor class and was using that whenever needed.
During the test I got one error:
This BackgroundWorker is currently busy and cannot run multiple tasks
concurrently
I tried to find out the solution and many people suggest to use new instance for each task. I changed my code like:
...
using (BackgroundWorker bw = new BackgroundWorker())
{
bw.DoWork += new DoWorkEventHandler(bkDoPost);
bw.RunWorkerAsync(dbobj);
}
...
and my 'bkDoPost' is:
void bkDoPost(object sender, DoWorkEventArgs e)
{
try
{
dbObject dbj = e.Argument as dbObject;
this.db.Insert(dbj.tableName, dbj.data);
}
catch (Exception ex)
{
logs.logMessage("There was an error in data post. See the ErrorLog");
logs.logError(ex);
}
}
The code works fine during test.
My question is am I doing correct way?
OR Is there any issue doing in that way?
Thanks

Don't do that. Your background worker will be disposed before your work completes.
It is better to call Dispose manually after the work completes.
Better still, consider using a different scheme for handling asynchronous work. Background worker is becoming obsolete and is targeted at UI applications, rather than services. The restriction on parallel operations highlights the intention of the class.

Don't put the BackgroundWorker into a using statement. Instead put the Dispose() call into the RunWorkerCompleted event.
Nevertheless BackgroundWorker is maybe not the best thing to use in your case, cause it is primilary use is to run some buisness code while the UI stays responive and to automatically update the UI within the RunWorkerCompeleted event.
If you don't need to interfere with the UI when the job is finished or you have a lot of smaller jobs to be done it would be more efficient to switch to encapsulate your jobs within Tasks.

If you have many updates, creating one BackgroundWorker for each one could be very time and memory consuming.
I would use an independant thread that I would wake up each time an update has to be done :
Queue<DbWork> dbUpdates = new Queue<DbWork>();
EventWaitHandle waiter = new EventWaitHandle(false, EventResetMode.ManualReset);
...
// Init :
new Thread(new ThreadStart(DbUpdateWorker));
...
private void DbUpdateWorker()
{
while (true)
{
DbWork currentWork = null;
lock (dbUpdates)
{
if (dbUpdates.Count > 0)
currentWork = dbUpdates.Dequeue();
}
if (currentWork != null)
{
currentWork.DoWork();
}
if (dbUpdates.Count == 0)
{
waiter.WaitOne();
}
}
}
public void AddWork(DbWork work)
{
lock (dbUpdates)
{
dbUpdates.Enqueue(work);
}
waiter.Set();
}

BeginInvoke is blocking the UI, whereas Invoke is not. Why?

I am confused with scenario which I have encountered with cross thread access. Here is what I am trying to do:
Main UI thread - menu item click I create a background worker and run it asynchronously
private void actionSubMenuItem_Click(object sender, EventArgs e)
{
ToolStripMenuItem itemSelected = (ToolStripMenuItem)sender;
ExecuteTheActionSelected(itemSelected.Text);
}
The method ExecuteTheActionSelected is as follows:
private void ExecuteTheActionSelected(string actionSelected)
{
BackgroundWorker localBackgroundWorker = new BackgroundWorker();
localBackgroundWorker.DoWork += new DoWorkEventHandler(localBackgroundWorker_DoWork);
localBackgroundWorker.RunWorkerAsync(SynchronizationContext.Current);
}
The localBackgroundWorker_DoWork has:
ActionExecutionHelper actionExecutioner = new ActionExecutionHelper()
actionExecutioner.Execute();
The Execute method in that class that has method invoker which infact invokes the event handler in UI thread:
public void Execute()
{
// ---- CODE -----
new MethodInvoker(ReadStdOut).BeginInvoke(null, null);
}
protected virtual void ReadStdOut()
{
string str;
while ((str = executionProcess.StandardOutput.ReadLine()) != null)
{
object sender = new object();
DataReceivedEventArgs e = new DataReceivedEventArgs(str);
outputDataReceived.Invoke(sender, e);
//This delegate invokes UI event handler
}
}
The UI event handler is as follows:
private void executionProcess_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
if (_dwExecuteAction != null)
{
_dwExecuteAction.ShowDataInExecutionWindow(e.Text);
}
}
Now here comes the cross thread issue:
public void ShowDataInExecutionWindow(string message)
{
if (rchtxtExecutionResults.InvokeRequired)
{
rchtxtExecutionResults.Invoke(new ShowDataExecutionDelegate(ShowDataInExecutionWindow), message);
}
else
{
this.rchtxtExecutionResults.AppendText(message + Environment.NewLine);
}
}
Here Invoke doesn't block the UI where as BeginInvoke blocks.
Please help me understand this scenario as i m confused a lot.

Yes, this is normal. The benefit you get out of Invoke() is that it blocks the worker thread. When you use BeginInvoke() the thread keeps motoring and issues invoke requests at a rate higher than the UI thread can handle. It depends on what you ask the UI thread to do but it starts to become a problem around 1000 invokes per second.
The UI thread stops being responsive in this scenario, it is constantly finding another invoke request back while it pumps the message loop and doesn't get around doing its regular duties anymore. Input and paint requests no longer get processed.
The clear source of the problem is the invoke request on every single line of output retrieved from the process. It is just generating them too quickly. You need to fix this by lowering the rate at which you invoke. There's a simple rule for that, you are only trying to keep a human occupied, invoking more than 25 times per second turns whatever you produce in but a blur to the eye. So buffer the lines and measure the amount of time that has passed since the last invoke call.
Also note that using Invoke() is an easy workaround but it isn't exactly guaranteed to work. It is a race, the worker thread could potentially always call the next Invoke() a wee bit earlier than the main thread re-entering the message loop and reading the next message. In which case you will still have the exact same problem.

Availability of windows form during processing

I'm doing an application that does some sort of scanning (it checks availability of URL's through a short list) and depending on the result, it adds to one or another listbox. if it exists, it goes to lstOK, else, it goes to lst404.
The issue is that these web checks take time (specially when it is OK), it takes an awfully long time, and inserts all the items in the listboxes in the end, while the form is "not responding" and nothing appears or can be clicked or displays any interaction.
Is there a way for the form to be still usable and the listboxes to update on the go ?
This should be simple, I just don't know it (yet)
I'm using C# in Visual Studio
--[update]--
The whole url checking is in one single function Start()

try the background worker

If this is a desktop application that is performing these "web checks" then you can use a BackgroundWorkerThread to perform the processing, and get the results.
Or you could do something like this:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.Threading;
namespace ThreadWithDataReturnExample
{
public partial class Form1 : Form
{
private Thread thread1 = null;
public Form1()
{
InitializeComponent();
thread1 = new Thread(new ThreadStart(this.threadEntryPoint));
Thread1Completed += new AsyncCompletedEventHandler(thread1_Thread1Completed);
}
private void startButton_Click(object sender, EventArgs e)
{
thread1.Start();
//Alternatively, you could pass some object
//in such as Start(someObject);
//With apprioriate locking, or protocol where
//no other threads access the object until
//an event signals when the thread is complete,
//any other class with a reference to the object
//would be able to access that data.
//But instead, I'm going to use AsyncCompletedEventArgs
//in an event that signals completion
}
void thread1_Thread1Completed(object sender, AsyncCompletedEventArgs e)
{
if (this.InvokeRequired)
{//marshal the call if we are not on the GUI thread
BeginInvoke(new AsyncCompletedEventHandler(thread1_Thread1Completed),
new object[] { sender, e });
}
else
{
//display error if error occurred
//if no error occurred, process data
if (e.Error == null)
{//then success
MessageBox.Show("Worker thread completed successfully");
DataYouWantToReturn someData = e.UserState as DataYouWantToReturn;
MessageBox.Show("Your data my lord: " + someData.someProperty);
}
else//error
{
MessageBox.Show("The following error occurred:" + Environment.NewLine + e.Error.ToString());
}
}
}
#region I would actually move all of this into it's own class
private void threadEntryPoint()
{
//do a bunch of stuff
//when you are done:
//initialize object with data that you want to return
DataYouWantToReturn dataYouWantToReturn = new DataYouWantToReturn();
dataYouWantToReturn.someProperty = "more data";
//signal completion by firing an event
OnThread1Completed(new AsyncCompletedEventArgs(null, false, dataYouWantToReturn));
}
/// <summary>
/// Occurs when processing has finished or an error occurred.
/// </summary>
public event AsyncCompletedEventHandler Thread1Completed;
protected virtual void OnThread1Completed(AsyncCompletedEventArgs e)
{
//copy locally
AsyncCompletedEventHandler handler = Thread1Completed;
if (handler != null)
{
handler(this, e);
}
}
#endregion
}
}

If it's a web form look into AJAX.NET. There are several controls (UpdatePanel being one off the top of my head) that will help you do this.
Take a look at the toolkit.
EDIT: Only for web apps.

Application.DoEvents(); will do all the events that have happened up to that point.
so in your loop, after each website is checked, for example. do Application.DoEvents();
on the other hand if you just want to refresh your listboxes it'll be listboxname.Refresh();
both of these options, however will still have a time where it freezes while the website is pinged, unless you do many of them, which i dont suggets doing.
both methods also only use a single thread and is very linear.
The best option would be to create a new thread to do the tests on, or use a background worker that can do the tests on a seperate thread, so the events of the form can be handled instantly without a need to wait.
Manually controlling another thread shouldnt be too difficult.
here's an example.
using System.Threading;
public class MultiThreadingClass
{
private void FunctionForNewThread()
{
//do stuff
}
private void FunctionWithParameter(object param)
{
//Should do checks with typeof() on param before casting
int convertedparam = (int)param;
//do stuff
}
Thread t, t2;
static void Main()
{
ThreadStart ts = new ThreadStart(FunctionForNewThread);
t = new Thread(ts);
t.Start();
int x = 5;
ParameterizedThreadStart pts = new ParameterizedThreadStart(FunctionWithParameter);
t2 = new Thread(pts);
t2.Start(x);
}
}
it may be important to note here that you should never add a Thread as a local variable that will dissapear, as you can only really get the thread instance back by doing Thread.CurrentThread in the function which was called by the new thread, but if that thread has already locked up, you have a bit of a problem there :)
To easily handle Threads in a global variable either create an Array of threads and call Thread.Abort(); on each running thread when the program closes, or use the ThreadPool class in System.Threading.

Wrapping an asynchronous method synchronously in C#

I have a third party library containing a class which performs a function asynchronously. The class inherits from the Form. The function basically performs a calculation based on data stored in a database. Once it has finished, it calls a _Complete event in the calling form.
What I would like to do is call the function synchronously but from a non-windows form application. The problem is, no matter what I do, my application blocks and the _Complete event handler never fires. From a windows form I can simulate the function running synchronously by using a "complete" flag and a "while (!complete) application.doevents", but obviously application.doevents isnt available in a non-windows form application.
Is there something that would stop me using the class's method outside of a windows form application (due to it inheriting from 'Form') ?
Is there some way I can work around this ?
Thanks,
Mike

At a stab it might be worth trying something like the following which uses a WaitHandle to block the current thread rather than spinning and checking a flag.
using System;
using System.Threading;
class Program
{
AutoResetEvent _autoEvent;
static void Main()
{
Program p = new Program();
p.RunWidget();
}
public Program()
{
_autoEvent = new AutoResetEvent(false);
}
public void RunWidget()
{
ThirdParty widget = new ThirdParty();
widget.Completed += new EventHandler(this.Widget_Completed);
widget.DoWork();
// Waits for signal that work is done
_autoEvent.WaitOne();
}
// Assumes that some kind of args are passed by the event
public void Widget_Completed(object sender, EventArgs e)
{
_autoEvent.Set();
}
}

I've got some more information on this problem (I'm working in the same team as mikecamimo).
The problem also occurs in the Windows Forms application, when replicated correctly. In the original OP, the problem didn't occur in the windows form because there was no blocking. When blocking is introduced by using a ResetEvent, the same problem occurs.
This is because the event handler (Widget_Completed) is on the same thread as the method calling Widget.DoWork. The result that AutoResetEvent.WaitOne(); blocks forever because the event handler is never called to Set the event.
In a windows forms environment this can worked around by using Application.DoEvents to poll the message queue and allow the event the be handled. See below.
using System;
using System.Threading;
using System.Windows.Forms;
class Program
{
EventArgs data;
static void Main()
{
Program p = new Program();
p.RunWidget();
}
public Program()
{
_autoEvent = new AutoResetEvent(false);
}
public void RunWidget()
{
ThirdParty widget = new ThirdParty();
widget.Completed += new EventHandler(this.Widget_Completed);
data = null;
widget.DoWork();
while (data == null);
Application.DoEvents();
// do stuff with the results of DoWork that are contained in EventArgs.
}
// Assumes that some kind of args are passed by the event
public void Widget_Completed(object sender, EventArgs e)
{
data = e;
}
}
In a non windows forms application, such as a Windows Service, Application is not available so DoEvents cannot be called.
The problem is one of threading and that widget.DoWork's associated event handler somehow needs to be on another thread. This should prevent AutoResetEvent.WaitOne from blocking indefinitely. I think... :)
Any ideas on how to accomplish this would be fantastic.

AutoResetEvent _autoEvent = new AutoResetEvent(false);
public WebBrowser SyncronNavigation(string url)
{
WebBrowser wb = null;
wb = new WebBrowser();
wb.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(wb_DocumentCompleted);
wb.ScriptErrorsSuppressed = true;
wb.Navigate(new Uri(url));
while (!_autoEvent.WaitOne(100))
Application.DoEvents();
return wb;
}
void wb_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
//throw new NotImplementedException();
_autoEvent.Set();
}

Do you have the source for the component? It sounds like it's relying on the fact it will be called from a WinForms environment (must be a good reason why a library inherits from Form!), but it's hard to know for sure.