We Keep Coding

C#/.NET: Reporting subprocess state to the parent service

I currently have a service running several subprocesses (with System.Diagnostics.Process). Each subprocess can run for hours and be in a specific, predefined state (think "starting", "working", "cleaning up", etc - completely predefined, no custom data attached to each state has to be reported). Each process cannot be an individual Windows service (there are more possible states than Windows service states). I need to somehow report this state to the parent service. All processes are running on the same Windows machine.
I need to be able to both query subprocess states from other processes (not the ones started by the service), and update the parent service about each subprocess states from those subprocesses. Each process gets a unique ID, so other processes can read the states easily without having to manage processes themselves. All processes share a configuration file in which each subprocess gets assigned a unique ID to identify itself with. I've thought about doing it like so:
Redirect subprocesses' standard output to the service (RedirectStandardOutput = true), read each line in the output and catch "special" lines (STATECHANGE:state)
Write out all subprocesses' states to a file in a predefined location whenever that state changes, delete that file on service exit.
It looks like I'm trying to find a solution to a problem which was solved ages ago and I haven't found that solution. Is there any "nice" way to do such state reporting?

In general, you're delving into the realm of interprocess communications, or IPC.
Though you haven't tagged this question as being specific to Microsoft Windows, it is tagged as C# and .NET, so it's probably that you are running in a Windows environment. My answer assumes you're running this system in a MS Windows.
A common solution to a problem such as this is store state in database. Each service/process could write to the database independently, and then it could be queried by any process that was interested in that information. But this isn't real two-way communication.
Regarding how the parent could communicate with the child processes, this could be done a number of ways, but it would probably be easiest if the child process ran some kind of message pump on a thread and performed data processing on another thread. The message pump would receive and respond to messages, while the data processing thread would do its thing.
Using this scheme, messages could be exchanged in a number of different ways, including:
Windows Communications Framework (WCF)
Named Pipes
.NET Remoting
MS Message Queue (MSMQ)
Windows Clipboard
Dynamic Data Exchange (DDE)
Component Object Model (COM)
Memory-mapped Files
Remote Procedure Calls (RPC)
Sockets
Since all of these processes are running on the same machine, pipes are a simple straightforward choice. Check out the System.IO.Pipes namespace
WCF allows you to build a rich messaging interface that can be implemented on top of pipes, as well as on top of other IPC mechanisms.
There are lots of good resources on the I'net that discuss interprocess communications on .NET, and rather than rehash those here, you should search these out using terms such as ".NET", "interprocess communications", "IPC" and "local machine" (since you need IPC between processes on the local machine).

Pass informations between separate consoles and windows applications

I have two separate programs, one is a console application, and the other one is a windows application.
My windows application:
Has a graphic interface, buttons, and others functions.
One of the buttons, named "research": when I click on it, I launch the console application with this line of code:
string strResult = ProcessHelper.LaunchProcessWaitForPipedResult("MyExecFile.exe", strArguments, 10 * 60 * 1000, true); // 10 mins max
My console Application:
do a query on all existing files in a directory.
My problem:
I want to create a progress-bar on the windows application to show the progress of the console application. The problem is I don't know how to pass this information between the two processes. The only restriction is to not use a database or file.

Given two processes in the same user session, and wanting to avoid any communication outside that session I would look at three options:
1. Using named pipes.
The parent process creates a named pipe using a random name (and confirms that name is not in use by opening it). It passes that name to the child process. A simple protocol is used that allows the child to send updates.
There are a number of challenges to overcome:
Getting the logic to ensure the name is unique right (named pipe names are global).
Ensuring no other process can connect (the default named pipe ACL limits connections to the session: this might be enough).
Handling the case where a different parent process does not support progress updates.
Handling the child or parent crashing.
Avoiding getting too clever with the communication protocol, but allowing room for growth (what happens when more than a simple progress bar is wanted?)
2. Using Shared Memory
In this case names of objects are, by default, local to the session. By default this is more secure.
The parent process creates a sufficiently large amount of shared memory (for a simple progress update: not much), a mutex and an event.
The parent process then, concurrently with the GUI waits for the event to be signalled, when it is it enters the mutex and reads the content of shared memory. It then unsets the event and leaves the mutex.
Meanwhile to send an update the child enters the mutex, updates and memory and sets the event before leaving the mutex.
The challenges here include:
Defining the layout of the shared memory. Without a shared assembly this is likely to be error prone.
Avoiding others using the shared memory and synchronisation objects. .NET makes things harder here: in Win32 I would make the handles inheritable thus not needing to name the objects (except for debugging) and pass to the child directly.
Getting the sequencing of shared memory, mutex and event correct is critical. Memory corruption and more subtle bugs await any errors.
It is harder to do variable length data with shared memory, not an issue for a simple progress count but customers always want more.
Summary
I would probably look at named pipes in the first place (or perhaps custom WMI types if I wanted greater flexibility). BUT I would do that only after trying everything to avoid needing multiple processes in the first place. A shared library plus console wrapper for others, while I use the library directly would be a far easier option.

Is there a way to associate arbitrary data to a Windows Process?

I have many instances of a process I've written on a server. I'd like to associate some information with each process. In this specific case I'd like to store the "CurrentState" of the process - "RUNNING|DRAINING|STOPPING", but it would be useful for me to store a "Friendly Name" and so on.
I want to query this information from another "mother" process - this mother process will query the processes running and collate the data.
I've thought of a couple of different ways I could achieve this. For example I might open up a NetPipe to each process of interest and ask for the data, or have each process broadcast it's state regularly.
I was wondering: is there a way to store key value pair information against a process built into Windows itself? Is there an accepted pattern for doing this?
I control the source for the child processes and the mother process. They are written in C#, P/Invoking is fine. The operating system is Windows 2012 R2.

You can host WCF services that use named pipes:
http://msdn.microsoft.com/en-us/library/ms733769(v=vs.110).aspx
Based on some of your comments, it looks like you could also consider the System.AddIn (aka Managed AddIn Framework (MAF)) functionality to create, host, and communicate with Add-ins. MAF supports loading addins in your app domain, a separate app domain, or in a completely separate process. The downside with MAF is that it requires 5 DLLs to get started, but in doing that gives you a lot of flexibility with API compatibility as you version and change your pipeline.

If you're controlling the data from a Mother process, you can also use AppDomains to load your other processes and communicate via Marhsaled data such as a Status class, or use the AppDomains to Set and Get data.
Be aware that any Status data you transfer needs to either be a class which derives from the Marshaling class or be marked as Serializable. The reason for this is because AppDomains are treated in the OS the same as different processes, so they can't access each others memory an actually have to serialize data as if it were being passed through IPC.

Take a look at the .Net Process Class:
http://msdn.microsoft.com/en-us/library/system.diagnostics.process(v=vs.110).aspx
You can use it to get all running processes, start a process, get the processes unique Id, and be alerted when the process exits. This should give you everything you need to track processes.
Children can call Process.GetCurrentProcess to get their own process id, then make a call to the "mother" process to associate arbitrary data about itself.

Synchronize multiple processes to begin working at the same time?

I'm working in .NET 4 in C#. I have LauncherProgram.exe that will create multiple instances of WorkerProgram.exe, have them do some work on the arguments supplied when the process is created, and then LauncherProgram.exe will launch a new set of WorkerProgram.exe instances to do some different work.
Each WorkerProgram.exe is launched with some parameters that tell it what to work on, and there can be one or more WorkerProgram.exe launched at the same time. The WorkerProgram.exe reads the supplied parameters, performs some initialization, and then is ready to do the work.
What I'm trying to figure out is how to make each set of WorkerProgram.exe launched at the same time "tell" or "signal" or "I can't figure out the proper term" the LauncherProgram.exe that EACH process has completed the initialization step and is ready to begin. I want to synchronize the start of the "do your work" in the WorkerProgram.exe instances launched in a set.
I'm setting up my LauncherProgram.exe to work something like this (ignoring types for now):
while (there are sets of work to do)
{
for each set of work
{
for each group data in set
create and launch a WorkerProgram.exe for a single set of data
wait for all created WorkerProgram.exe to indicate init is complete
send signal to start processing
}
}
I actually have a small test program where I use named events to signal multiple spawned processes to START something at the same time.
(Hopefully all the above makes sense)
I just can't figure out the "wait for N processes to tell me their initialization is ready" bit.
I've searched for "process synchronization" and "rendezvous" and looked at using named events and named semaphores. I can find a bunch of things about threads, but less about separate processes.
LauncherProgram.exe creates the WorkerProgram.exe processes using the System.Diagnostics.Process class, if that helps.
If you can give me better terms to help narrow my search, or point me to a design pattern or mechanism, or a library or class that helps, I'd be very appreciative.
Thanks.

You can use the System.Threading.Mutex class for interprocess communication. See http://msdn.microsoft.com/en-us/library/system.threading.mutex(v=vs.110).aspx. It is probably easiest to name each Mutex, giving the process id of WorkerProgram.exe or some other distinguishing characteristic as the name.

You can use some interprocess communication but the simple way to do it is to write to temp files for instance writing DONE to some file and having Launcher read periodically until all WorkerProgram write DONE to their respective files, etc... or even create a FileMapping in windows to share memory between processes with file backings.
Other ways to do it include remote procedure calls, sockets, and simple file mappings.

How to query a variable in another running application in c#?

I have an app that, when launched, checks for duplicate processes of itself.
That part I have right - but what I need is to check a state variable in the original running process in order to run some logic.
So: how do I make a variable (e.g. bool) available publicly to other applications so they can query it?

There are a bunch of ways to do this. A very primative way would be to read/write from a file. The old win32 way would be to use PostMessage. The more .NET way would be to use remoting or WCF and Named Pipes.
.NET 4 is also getting support for Memory Mapped files.
Here is a pretty thorough looking artcile describing a few different approaches including support for Memory Mapped files outside of .NET 4
http://www.codeproject.com/KB/threads/csthreadmsg.aspx

The easiest: Create a file, and write something in it.
More advanced, and when done correctly more robust, is using WCF, you use named pipes to setup some communication channel on the local computer only.

If you're using a Mutex to check whether another process is running (you should be) you could use another Mutex whose locked state would be the boolean flag you're looking.

The standard way of doing this is to use the Windows API to create and lock a mutex. The first app to open will create and lock the mutex. Any subsequent executions of the app will not be able to get it and can then shutdown.