We Keep Coding

Choosing the right pattern to expose different kinds of services (RR, PP, PS) from a proxy to N clients

I am working with a robot from the manufacturer Kuka and my goal is :
to expose some of the functionalities of the robot system to remote clients through a clients-proxy architecture.
Hereafter I give a description of the system I want to build. And at the bottom there is my question(s).
Description of the robot
The robot is composed of a Controller hosted on a WIN XP embedded system, running on a standard PC. The Controller drives the arm, the track and a I/O bus. It is responsible for running programs written in the Kuka Robot Language (KRL) and this is the prescribed way of interacting with the robot cell.
XComm interface
Kuka provides the XComm interface to interact with the robot controller from the WIN XP system. It is compatible with .Net Framework 4.0. This interface is divided into services, and each service comes in two flavors : blocking call vs. async call with callback.
For instance the XComm.Var service is mainly exposing these Sync/Async interfaces :
interface SyncVar
{
public string GetVar(string varName); // reads a variable
public void SetVar(string varName, string varValue); // changes a variable
}
interface AsyncVar
{
// reads a variable
public void GetVarAsync(string varName, AsyncCallback callback);
// changes a variable
public void SetVarAsync(string varName, string varValue, AsyncCallback callback);
// subscribe to varValueChanged event : callback will be invoked each time the monitored variable has changed
public void InfoVar_ON(string varName, AsyncCallback callback, Timespan ts);
// unsubscribe to varValueChanged event
public void InfoVar_OFF(string varName);
}
There is a dozen of such services : XComm.Motion, XComm.IO, XComm.File, ...
Objectives
The main feature I am concerned with is interacting remotely with the robot as a state machine. That is I want a remote client to be able to get the state of some variables and potentially change the state of some variables at any moment.
Additionally, I would be interested (latter) to expose other XComm services like the XComm.File for loading KRL programs to the robot and XComm.Module to run them from a remote client. So I need my architecture to be extensible to other XComm services.
Features
1) REQ-REP
The first way I would like a client to communicate with the robot is through a simple REQ-REP pattern. For instance :
C : Hey robot, what si the speed of your arm now ?
S : It is 10m/s
C : Ok, and what is the torque on your tool ?
S : It is 1000Nm
C : Oh, I see, the stone is harder than I expected, please reduce your speed now to 5m/s
...
Or :
C : Hey robot, please, move to x = 1000mm at speed 1000mm/s
S : Ok client, I will move to x = 1000mm
C (+250ms) : Hey robot, where are you now ?
S : I am at x = 230mm
C (+500ms) : Hey robot, where are you now ?
S : I am at x = 550mm
C (+750ms) : Hey robot, where are you now ?
S : I am at x = 800mm
C (+1000ms) : Hey robot, where are you now ?
S : I am at x = 1000mm
C : Ok, so now please move to y = -500mm at speed 250mm/s
...
2) PUSH-PULL
The second way I would like a client to communicate with the robot is through a simple PUSH-PULL pattern where clients can subscribe to get notified when the value of a variable has changed. Something which could look like :
C : Hey robot, let me know when you move the arm on the track, but not more than every 50ms
S : Ok client, I will let you know
...
S : Hey client, I've just move the arm and the track an my position is now E1=4000mm
...
S : Hey client, I've just move the arm and the track an my position is now E1=4010mm
...
C : Hey robot, I'm done with all this, stop notifying my when you move the arm on the track.
S : Ok client, understood.
3) PUB-SUB
The third way I would like a client to communicate with the robot is through a simple PUB-SUB pattern. I want a client to be able to subscribe to some predefined channels (ie. motion, command, IO). A channel broadcasts the state of a set of related variables as a list ok key-value pairs. This broadcast occurs every 40ms or so and is mainly meant to build dashboards and refresh them at more or less 25fps. The configuration of the channels is static but might be loaded from a configuration file when the proxy starts.
Performance
Latency
A (local) synchronous call to a XComm service is about 5ms long. It is ok but yet too much (by 5 to 10) for certain real time control applications. So I really need to be cautious that the chosen architecture will not increase this latency. So obviously I need my clients to send async requests to no cumulate the network roundtrip time of each REQ-REP cycle.
Maybe dispatching client requests through several XComm service consumers with a load balancing pattern could help reducing the latency ?
Throughput
When I am concerned with low latency, it is always for messages with small footprints for reading and writing variable states (few hundred bytes max).
Let us say I want to drive the robot position at 1000fps. That is :
2 ways * 256 bytes / msg * 1000 msgs/sec = 512kB/s = 4.096Mbs
So clearly, throughput is not a problem on a basic 1Gbs LAN.
If I need to load a KRL program on the robot, I can afford to wait few 100ms or sec to transfer the file.
Reliability
I need my REQ-REP pattern to be reliable to network, server and client failures and to recover when I manually restart a server or a client. So I need a PING-PONG keep alive.
Possibly I need a "Secure" version of the Get/Set variable service when, for instance, I want to stream positions to the robot so I can resume my streaming at the point where it stopped. Something like the titanic pattern with a persistant queue somewhere on the disk ?
Architecture / Pattern
So my first bet is that a Majordomo pattern will be adapted to my situation.
1 Proxy running on WIN XP
workers : each worker acts as a consumer for a single XComm service. It can consume sync or async.
broker : a router frontend with a dealer backend that dispatches the client requests to the worker and the worker replies back to the clients.
discovery : service for clients to known which services are available from the proxy and their state.
N < 10 Clients running remotely on the same LAN or over the internet
clients can come and go at any time
a client can Get/Set one or several variables with a single request (possibly with a safe option) and in the flavor of their choice (sync/async).
For Async Get/Set var I need a per request callback mechanism : when the reply has arrived, the callback is invoked so the client as a built-in way of triggering and dispatching actions to be done with the replies.
a client can Subscribe/Unsubscribe to variable monitoring (push notification)
a client can Subscribe/Unsubscribe to a predefined channel
a client can discover the available services from the proxy and know their state
a client can discover which clients are connected to the proxy and their state
I want to provide a simple client API that any device will have to implement to consume the services from the proxy.
Ideally, I would like one ZSocket per client and one ZSocket for the proxy frontend to have a simpler heartbeat keepalive mechanism.
The remote client-proxy communications will TCP.
Potentially, local client-proxy is allowed through inproc
The proxy-worker communications will be inproc (faster right ?).
Questions
So I have identified 3 services that my proxy should provide to clients :
REQ-REPL to Get/Set robot variables
PUSH-PULL to Subscribe/Unsubscribe to get notified when a certain variable has changed
PUB-SUB to receive (partial) snapshots of the state of the robot at regular intervals
(DISCOVERY) to know the available services
=> So my question is what's the right pattern for my clients-proxy so they talk to each others consuming (potentially) the 3 services at the same time ?
=> Do I need to provide 3 frontend/ports/endpoints, one for each service ?
=> Can I have a unique frontend dispatching to each service and routing REQ/REP/PUSH/PULL/PUB/SUB messages to the right clients ?
I feel I prefer to keep a unique frontend socket on my proxy talking to clients with a unique socket too. I feel it will be easier to manage reliability this way rather than if i need a socket per service (3 heartbeats PING/PONG). And my client will stay minimal. But I might be completely wrong.

WCF Windows Service - Long operations/Callback to calling module

I have a Windows Service that takes the name of a bunch of files and do operations on them (zip/unzip, updating db etc). The operations can take time depending on size and number of files sent to the service.
(1) The module that is sending a request to this service waits until the files are processed. I want to know if there is a way to provide a callback in the service that will notify the calling module when it is finished processing the files. Please note that multiple modules can call the service at a time to process files so the service will need to provide some kind of a TaskId I guess.
(2) If a service method is called and is running and another call is made to the same service, then how will that call be processed(I think there is only one thread asociated with the service). I have seen that when the service is taking time in processing a method, the threads associated with the service begin to increase.

WCF does indeed offer duplex bindings which allow you to specify a callback contract, so that the service can call back to the calling client to notify.
However, in my opinion, this mechanism is rather flaky and not really to be recommended.
In such a case, when the call causes a fairly long running operation to happen, I would do something like this:
If you want to stick to HTTP/NetTcp bindings, I would:
drop off the request with the service, and then "let go" - this would be a one-way call, you just drop off what you want to have done, and then your client is done
have a status call that the client could call after a given time to find out whether or not the results of the request are ready by now
if they are, there should be a third service call to retrieve the results
So in your case, you could drop off the request to zip some files. The service would go off and do its work and store the resulting ZIP in a temporary location. Then later on the client could check to see whether the ZIP is ready, and if so, retrieve it.
This works even better over a message queue (MSMQ) which is present in every Windows server machine (but not a lot of people seem to know about it or use it):
your client drops off the request on a request queue
the service listens on that request queue and fetches request after request and does it works
the service can then post the results to a result queue, on which your callers in turn are listening
Check out how to do all of this efficiently by reading the excellent MSDN article Foudnations: Build a queue WCF Response Service - highly recommended!
A message-queue based systems tends to be much more stable and less error-prone that a duplex-/callback-contract based system, in my opinion.

(1) The simplest way to achieve this is with a taskId as you note, and then have another method called IsTaskComplete with which client can check whether the task has been completed.
(2) Additional calls made to the service will start new threads.
edit: the default service behaviour is to start new threads per call. The configurable property is Instance Context Mode, and can be set to PerCall, PerSession, or Shareable.

The question has a solution, but I'm using a WCF duplex service to get the result of a long operation, and even though I found a problem that has cost me several hours to solve (and that's why I searched this question earlier), now it works perfectly, and I believe it is a simple solution within the WCF duplex service framework.
What is the problem with a long operation? The main problem is blocking the client interface while the server performs the operation, and with the WCF duplex service we can use a call back to the client to avoid the blockage (It is an old method to avoid blocking but it can easily be transformed into the async/await framework using a TaskCompletionSource).
In short, the solution uses a method to start the operation asynchronously on the server and returns immediately. When the results are ready, the server returns them by means of the client call back.
First, you have to follow any standard guide to create WCF duplex services and clients, and I found these two useful:
msdn duplex service
Codeproject Article WCF Duplex Service
Then follow these steps adding your own code:
Define the call back interface with an event manager method to send results from the server and receive them in the client.
public interface ILongOperationCallBack
{
[OperationContract(IsOneWay = true)]
void OnResultsSend(....);
}
Define the Service Interface with a method to pass the parameters needed by the long operation (refer the previous ILongOperationCallBack interface in the CallBackContractAttribute)
[ServiceContract(CallbackContract=typeof(ILongOperationCallBack))]
public interface ILongOperationService
{
[OperationContract]
bool StartLongOperation(...);
}
In the Service class that implements the Service Interface, first get the proxy of the client call back and save it in a class field, then start the long operation work asynchronously and return the bool value immediately. When the long operation work is finished send the results to the client using the client call back proxy field.
public class LongOperationService:ILongOperationService
{
ILongOperationCallBack clientCallBackProxy;
public ILongOperationCallBack ClientCallBackProxy
{
get
{
return OperationContext.Current.GetCallbackChannel<ITrialServiceCallBack>());
}
}
public bool StartLongOperation(....)
{
if(!server.IsBusy)
{
//set server busy state
//**Important get the client call back proxy here and save it in a class field.**
this.clientCallBackProxy=ClientCallBackProxy;
//start long operation in any asynchronous way
......LongOperationWorkAsync(....)
return true; //return inmediately
}
else return false;
}
private void LongOperationWorkAsync(.....)
{
.... do work...
//send results when finished using the cached client call back proxy
this.clientCallBackProxy.SendResults(....);
//clear server busy state
}
....
}
In the client create a class that implements ILongOperationCallBack to receive results and add a method to start the long operation in the server (the start method and the event manager don't need to be in the same class)
public class LongOperationManager: ILongOperationCallBack
{
public busy StartLongOperation(ILongOperationService server, ....)
{
//here you can make the method async using a TaskCompletionSource
if(server.StartLongOperation(...)) Console.WriteLine("long oper started");
else Console.Writeline("Long Operation Server is busy")
}
public void OnResultsSend(.....)
{
... use long operation results..
//Complete the TaskCompletionSource if you used one
}
}
NOTES:
I use the bool return in the StartLongOperation method to indicate that the server is Busy as opposed to down, but it is only necessary when the long operation can't be concurrent as in my actual application, and maybe there are best ways in WCF to achieve non concurrency (to discover if the server is down, add a Try/Catch block as usual).
The important quote that I didn't see documented is the need to cache the call back client proxy in the StartLongOperation method. My problem was that I was trying to get the the proxy in the working method (yes, all the examples use the call back client proxy in the service method, but it isn't explicity stated in the documentation, and in the case of a long operation we must delay the call back until the operation ends).
Do not get and cache twice the call back Proxy after a service method has returned and before the next one.
Disclaimer: I haven't added code to control errors, etc.

WCF, BasicHttpBinding: Stop new connections but allow existing connections to continue

.NET 3.5, VS2008, WCF service using BasicHttpBinding
I have a WCF service hosted in a Windows service. When the Windows service shuts down, due to upgrades, scheduled maintenance, etc, I need to gracefully shut down my WCF service. The WCF service has methods that can take up to several seconds to complete, and typical volume is 2-5 method calls per second. I need to shut down the WCF service in a way that allows any previously call methods to complete, while denying any new calls. In this manner, I can reach a quiet state in ~ 5-10 seconds and then complete the shutdown cycle of my Windows service.
Calling ServiceHost.Close seems like the right approach, but it closes client connections right away, without waiting for any methods in progress to complete. My WCF service completes its method, but there is no one to send the response to, because the client has already been disconnected. This is the solution suggested by this question.
Here is the sequence of events:
Client calls method on service, using the VS generated proxy class
Service begins execution of service method
Service receives a request to shut down
Service calls ServiceHost.Close (or BeginClose)
Client is disconnected, and receives a System.ServiceModel.CommunicationException
Service completes service method.
Eventually service detects it has no more work to do (through application logic) and terminates.
What I need is for the client connections to be kept open so the clients know that their service methods completed sucessfully. Right now they just get a closed connection and don't know if the service method completed successfully or not. Prior to using WCF, I was using sockets and was able to do this by controlling the Socket directly. (ie stop the Accept loop while still doing Receive and Send)
It is important that the host HTTP port is closed so that the upstream firewall can direct traffic to another host system, but existing connections are left open to allow the existing method calls to complete.
Is there a way to accomplish this in WCF?
Things I have tried:
ServiceHost.Close() - closes clients right away
ServiceHost.ChannelDispatchers - call Listener.Close() on each - doesn't seem to do anything
ServiceHost.ChannelDispatchers - call CloseInput() on each - closes clients right away
Override ServiceHost.OnClosing() - lets me delay the Close until I decide it is ok to close, but new connections are allowed during this time
Remove the endpoint using the technique described here. This wipes out everything.
Running a network sniffer to observe ServiceHost.Close(). The host just closes the connection, no response is sent.
Thanks
Edit: Unfortunately I cannot implement an application-level advisory response that the system is shutting down, because the clients in the field are already deployed. (I only control the service, not the clients)
Edit: I used the Redgate Reflector to look at Microsoft's implementation of ServiceHost.Close. Unfortunately, it calls some internal helper classes that my code can't access.
Edit: I haven't found the complete solution I was looking for, but Benjamin's suggestion to use the IMessageDispatchInspector to reject requests prior to entering the service method came closest.

Guessing:
Have you tried to grab the binding at runtime (from the endpoints), cast it to BasicHttpBinding and (re)define the properties there?
Best guesses from me:
OpenTimeout
MaxReceivedMessageSize
ReaderQuotas
Those can be set at runtime according to the documentation and seem to allow the desired behaviour (blocking new clients). This wouldn't help with the "upstream firewall/load balancer needs to reroute" part though.
Last guess: Can you (the documention says yes, but I'm not sure what the consequences are) redefine the address of the endpoint(s) to a localhost address on demand?
This might work as a "Port close" for the firewall host as well, if it doesn't kill of all clients anyway..
Edit: While playing with the suggestions above and a limited test I started playing with a message inspector/behavior combination that looks promising for now:
public class WCFFilter : IServiceBehavior, IDispatchMessageInspector {
private readonly object blockLock = new object();
private bool blockCalls = false;
public bool BlockRequests {
get {
lock (blockLock) {
return blockCalls;
}
}
set {
lock (blockLock) {
blockCalls = !blockCalls;
}
}
}
public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
}
public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) {
}
public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) {
foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) {
endpointDispatcher.DispatchRuntime.MessageInspectors.Add(this);
}
}
}
public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) {
lock (blockLock) {
if (blockCalls)
request.Close();
}
return null;
}
public void BeforeSendReply(ref Message reply, object correlationState) {
}
}
Forget about the crappy lock usage etc., but using this with a very simple WCF test (returning a random number with a Thread.Sleep inside) like this:
var sh = new ServiceHost(new WCFTestService(), baseAdresses);
var filter = new WCFFilter();
sh.Description.Behaviors.Add(filter);
and later flipping the BlockRequests property I get the following behavior (again: This is of course a very, very simplified example, but I hope it might work for you anyway):
// I spawn 3 threads
Requesting a number..
Requesting a number..
Requesting a number..
// Server side log for one incoming request
Incoming request for a number.
// Main loop flips the "block everything" bool
Blocking access from here on.
// 3 more clients after that, for good measure
Requesting a number..
Requesting a number..
Requesting a number..
// First request (with server side log, see above) completes sucessfully
Received 1569129641
// All other messages never made it to the server yet and die with a fault
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request before the block.
Error in client request before the block.

Is there an api for the upstream firewall? The way we do this in our application is to stop new requests coming in at the load balancer level, and then when all of the requests have finished processing we can restart the servers and services.

My suggestion is to set an EventHandler when your service goes into a "stopping state", use the OnStop method. Set the EventHandler indicating that your service is going into a stopping state.
Your normal service loop should check if this event is set, if it is, return a "Service is stopping message" to the calling client, and do not allow it to enter your normal routine.
While you still have active processes running, let it finish, before the OnStop method moves on to killing the WCF host (ServiceHost.Close).
Another way is to keep track of the active calls by implementing your own reference counter. you will then know when you can stop the Service Host, once the reference counter hits zero, and by implementing the above check for when the stop event has been initiated.
Hope this helps.

I haven't implemented this myself, so YMMV, but I believe what you're looking to do is pause the service prior to fully stopping it. Pausing can be used to refuse new connections while completing existing requests.
In .NET it appears the approach to pausing the service is to use the ServiceController.

Does this WCF Service authenticate the user in any way?
Do you have any "Handshake" method?

I think you might need to write your own implementation with a helper class that keeps track of all running requests, then when a shutdown is requested, you can find out if anything is still running, delay shutdown based on that... (using a timer maybe?)
Not sure about blocking further incoming requests... you should have a global variable that tells your application whether a shutdown was requested and so you could deny further requests ...
Hope this may help you.

Maybe you should set the
ServiceBehaviorAttribute and the OperationBehavior attribute. Check this on MSDN

In addition to the answer from Matthew Steeples.
Most serious load balancers like a F5 etc. have a mechanism to identify if a node is alive. In your case it seems to check whether a certain port is open. But alternative ways can be configured easily.
So you could expose e.g. two services: the real service that serves requests, and a monitoring "heart beat"-like service. When transitioning into maintenance mode, you could first take the monitoring service offline which will take the load away from the node and only shutdown the real service after all requests finished processing. Sounds a bit weird but might help in your scenario...

WCF - Client callback vs. polling for "keep list of subscribers"

I want to create a simple client-server example in WCF. I did some testing with callbacks, and it works fine so far. I played around a little bit with the following interface:
[ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IStringCallback))]
public interface ISubscribeableService
{
[OperationContract]
void ExecuteStringCallBack(string value);
[OperationContract]
ServerInformation Subscribe(ClientInformation c);
[OperationContract]
ServerInformation Unsubscribe(ClientInformation c);
}
Its a simple example. a little bit adjusted. You can ask the server to "execute a string callback" in which case the server reversed the string and calls all subscribed client callbacks.
Now, here comes the question: If I want to implement a system where all clients "register" with the server, and the server can "ask" the clients if they are still alive, would you implement this with callbacks (so instead of this "stringcallback" a kind of TellTheClientThatIAmStillHereCallback). By checking the communication state on the callback I can also "know" if a client is dead. Something similar to this:
Subscribers.ForEach(delegate(IStringCallback callback)
{
if (((ICommunicationObject)callback).State == CommunicationState.Opened)
{
callback.StringCallbackFunction(new string(retVal));
}
else
{
Subscribers.Remove(callback);
}
});
My problem, put in another way:
The server might have 3 clients
Client A dies (I pull the plug of the laptop)
The server dies and comes back online
A new client comes up
So basically, would you use callbacks to verify the "still living state" of clients, or would you use polling and keep track "how long I havent heard of a client"...

You can detect most changes to the connection state via the Closed, Closing, and Faulted events of ICommunicationObject. You can hook them at the same time that you set up the callback. This is definitely better than polling.
IIRC, the Faulted event will only fire after you actually try to use the callback (unsuccessfully). So if the Client just disappears - for example, a hard reboot or power-off - then you won't be notified right away. But do you need to be? And if so, why?
A WCF callback might fail at any time, and you always need to keep this in the back of your mind. Even if both the client and server are fine, you might still end up with a faulted channel due to an exception or a network outage. Or maybe the client went offline sometime between your last poll and your current operation. The point is, as long as you code your callback operations defensively (which is good practice anyway), then hooking the events above is usually enough for most designs. If an error occurs for any reason - including a client failing to respond - the Faulted event will kick in and run your cleanup code.
This is what I would refer to as the passive/lazy approach and requires less coding and network chatter than polling or keep-alive approaches.

If you enable reliable sessions, WCF internally maintains a keep-alive control mechanism. It regularly checks, via hidden infrastructure test messages, if the other end is still there. The time interval of these checks can be influenced via the ReliableSession.InactivityTimeout property. If you set the property to, say, 20 seconds, then the ICommunicationObject.Faulted event will be raised about 20 to 30 (maximum) seconds after a service breakdown has occurred on the other side.
If you want to be sure that client applications always remain "auto-connected", even after temporary service breakdowns, you may want to use a worker thread (from the thread pool) that repeatedly tries to create a new proxy instance on the client side, and calls a session-initiating operation, after the Faulted event has been raised there.
As a second approach, since you are implementing a worker thread mechanism anyway, you might also ignore the Faulted event and let the worker thread loop during the whole lifetime of the client application. You let the thread repeatedly check the proxy state, and try to do its repair work whenever the state is faulted.
Using the first or the second approach, you can implement a service bus architecture (mediator pattern), guaranteeing that all client application instances are constantly ready to receive "spontaneous" service messages whenever the service is running.
Of course, this only works if the reliable session "as such" is configured correctly to begin with (using a session-capable binding, and applying the ServiceContractAttribute.SessionMode, ServiceBehaviorAttribute.InstanceContextMode, OperationContractAttribute.IsInitiating, and OperationContractAttribute.IsTerminating properties in meaningful ways).

I had a similar situation using WCF and callbacks. I did not want to use polling, but I was using a "reilable" protocol, so if a client died, then it would hang the server until it timed out and crashed.
I do not know if this is the most correct or elegant solution, but what I did was create a class in the service to represent the client proxy. Each instance of this class contained a reference to the client proxy, and would execute the callback function whenever the server set the "message" property of the class. By doing this, when a client disconnected, the individual wrapper class would get the timeout excetpion, and remove itself from the server's list of listeners, but the service would not have to wait for it. This doesn't actually answer your question about determining if the client is alive, but it is another way of structuring the service to addrss the issue. If you needed to know when a client died, you would be able to pick up when the client wrapper removed itself from the listener list.

I have not tried to use WCF callbacks over the wire but i have used them for interprocess communication. I was having a problem where call of the calls that were being sent were ending up on the same thread and making the service dead lock when there were calls that were dependant on the same thread.
This may apply to the problem that you are currently have so here is what I had to do to fix the problem.
Put this attribute onto the server and client of the WCF server implemetation class
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
public class WCFServerClass
The ConcurrencyMode.Multiple makes each call process on its own thread which should help you with the server locking up when a client dies until it timesout.
I also made sure to use a Thread Pool on the client side to make sure that there were no threading issues on the client side

How to check the availability of a net.tcp WCF service

My WCF server needs to go up and down on a regular basis, the client sometimes uses the server, but if it is down the client just ignore it.
So each time I need to use the server services I check the connection state and if it's not open I open it.
The problem is that if I attempt to open while the server is down there is a delay which hits performance.
My question is, is there a way to do some kind of myClient.CanOpen()? so I'd know if there is any point to open the connection to the server.

There is an implementation of WS-Discovery that would allow you to listen for up/down announcements for your service. This is also a very convenient form of service address resolution because it utilizes UDP multicast messages to find the service, rather than configuring one set address on the client.
WS-Discovery for WCF
There's also an implementation done by a Microsoft employee:
WS-Discovery Sample Implementation
.NET 4.0 will include this natively. You can read about .NET 4.0's implementation on Jesus Rodriguez's blog. It has a great chart that details the ad-hoc communication that goes on in WS-Disco Using WS-Discovery in WCF 4.0
Another thing you might consider, especially if your messages are largely one-way, is a protocol that works natively disconnected, like MSMQ. I don't know what your design for your application looks like, but MSMQ would allow a client to send a message regardless of the state of the service and the service will get it when it comes back up. This way your client doesn't have to block quite so much trying to get confirmation that a service is up before communicating... it'll just fire and forget.
Hope this helps.

If you are doing a synchronous call expecting a server timeout in an application with a user interface, you should be doing it in another thread. I doubt that the performance hit is due to exception overhead.
Is your performance penalty in CPU load, gui availability or wall clock time?
You could investigate to see if you can create a custom binding on TCP, but with faster timeout.
I assume you know that "IsOneWay=true" is faster than request->response in your case because you wouldn't be expecting a response anyway, but then you are not getting confirmation or return values.
You could also implement a two-way communication that is not request->response.

If you were in a local network it might be possible to broadcast a signal to say that a new server is up. The client would need to listen for the broadcast signal and respond accordingly.

Here's what I'm using and it works like a charm. And btw, the ServiceController class lives in namespace 'System.ServiceProcess'.
try
{
ServiceController sc = new ServiceController("Service Name", "Computer's IP Address");
Console.WriteLine("The service status is currently set to {0}",
sc.Status.ToString());
if ((sc.Status.Equals(ServiceControllerStatus.Stopped)) ||
(sc.Status.Equals(ServiceControllerStatus.StopPending)))
{
Console.WriteLine("Service is Stopped, Ending the application...");
Console.Read();
EndApplication();
}
else
{
Console.WriteLine("Service is Started...");
}
}
catch (Exception)
{
Console.WriteLine("Error Occurred trying to access the Server service...");
Console.Read();
EndApplication();
}

I don't think it's possible doing a server side call to your Client to inform him that you the service has been started ... Best method i can see is having a client method figuring out where or not the service is open and in good condition. Unless I am missing some functionality of WCF ...
There is a good blogpost WCF: Availability of the WCF services if you are interested in a read.