We Keep Coding

How to Stop Server Crash when Client Crashed internally (TCP/IP Socket)

Description about the problem.
Currently I am developing server - multiple clients (over 3000 clients) application in csharp. Server and clients exchange messages. The problem is that server get crashed if any of the client crashes for the internal reason. The internal reason can include all sort of specific problem of client computer. For example, they can be the lost internet connection, the computer crash, power failure, etc.
As the server functions to deliver the real time information, if the server crashes, then entire operation stops. This is nightmare for server because we can not predict which client will crash.
The main code causing the server crashes is below:
public bool Send(byte[] buffer)
{
if (m_Socket == null)
{
return (false);
}
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
return (false);
}
I think there were few answers to the similar problem from my research. However often the answers point out the use of Keep alive packets.
I think many answer in this website indicates that detecting the half open connection (dropped connection) is almost impossible without using keep alive packet from client side.
In our application, we do not prefer to use Keep alive packet or message because the server and client communicate in milliseconds. Receiving keep alive packet or message from over 3000 clients every seconds does not seems friendly for resource management point of view for server. So we prefer to have rather a good error management. If the client computer is crashed, then we just want to ignore. I am not sure whether this can make the running server unstable ?
Anyway, I came up with few ideas with below options. It was the suggestions from other coders most of time on the net. So the question is Which option might be the best to safely catch the Socket.Send() failure error and not disturbing the operation of the server?
Using simple try catch statement. Will this offer the 100% secure fail safe operation for this problem ? From my experience, sometimes, it works but sometimes it does not work.
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
Use of while(true) loop. Someone mentioned that while(true) loop might be the safe way of doing this. However, I am not sure how effective this solution comparing to simple try catch statement.
Perform Socket.Send() function inside other thread rather than main thread. The idea is that if Socket.Send() functions fails then the thread used for Socket.send() function will terminate on its own without disturbing the main thread for the application. However, I do not have sufficient knowledge how each thread suppose to interact together. So I can not tell whether this method is better than above two methods.
Can you please suggest any idea or solution for this specific problem? I think that without using Keep alive packet, maintaining the sever operation without crash is quite challenging problem.

C# - WCF Duplex => Fire an event when a client disconnect

I'm trying to let my service knows when one of the clients is disconnected.
I'm using wsDualHttpBinding.
Currently, I'm tried to use this event :
OperationContext.Current.Channel.Closed += new EventHandler((sender, e) => ClientIsDisconnected(sender, e, currentCallbackChannel));
But this event is never fired...
Please help me to know how it'd be done !
Edit :
Thanks to anderhil, I finally replaced wsDualHttpBinding by netTcpBinding (with the appropriate configuration described here : http://msdn.microsoft.com/en-us/library/ff647180.aspx#Step1).
With netTcpBinding, the Closed event fires without any problem... Still don't know why but it works.

The issue you are having is likely becuase of WsDualHttpBinding. In case you have this binding, two connections are created, from client to service and from service to client.
When the application is deployed over the internet it can create some issues with supporting such applications, you need to be sure that people are not behind the firewall or NAT or etc that can prevent your service to connect back to client.
I still don't know why it doesn't work on local machine when testing, but i will try to resolve it and update the answer.
As you told me more details in our chat, from the nature of your application it's better to use NetTcpBinding. In this case it's easier to understand what is happening cause one connection is created, and you will receive the notifications in case of gracefull close or abort of client.
As i told you before, anyway it's better to create some heartbeat mechanism to have things more reliable in case of unexpected computer or router shutdown.
Also, you can find this good cheat sheet on how to select communication between parties that involve WCF:

The Closed event should occur on a graceful disconnect; is that what's happening?
To detect the pure socket disconnect, listen for the Faulted event:
OperationContext.Current.Channel.Faulted += new EventHandler(FaultedHandler);

Problem about checking a WCF connection is opened

I have a problem about checking a WCF connection is opened. My WCF Connection is bi-directional. I use State property to check the connection's state at client. My function:
private bool ConnectionIsOpen()
{
if (m_Service != null && (m_Service.State | CommunicationState.Opened) == CommunicationState.Opened)
{
return true;
}
return false;
}
I create a service which is a thread running every 10 seconds to check the connection's state. I use the method ConnectionIsOpen() for checking. Everything is well on running on Windows XP. However there is a problem when running on Windows 7.
When I unplug the network cable to create to disconnect, If running application on Windows XP, checking connection's State is Faulted but if running on Windows 7, checking connection' State is still Opened.
Anyone can help me how to check a connection is openned or not in this case. Thanks.

This will always be true:
(m_Service.State | CommunicationState.Opened) == CommunicationState.Opened
Example, m_Service.State = 0:
0 | CommuncationState.Opened == CommuncationState.Opened
You want to use & (AND) instead.

We ran into a similar problem in our own system; disconnecting the network cable or placing either the client machine or the server in sleep mode does not generate a channel fault.
From what I can tell, it seems that the connection state only indicates the state of the connection after the last call and not the current connection state. The only way to know the current state is to actually call the service.
If your client doesn’t need to call the service that often but must react if the connection is lost one solution is to implement a dummy call on the client side which periodically polls the service. If the connection is unavailable when the dummy call is made you’ll get a channel fault that you can then deal with.
The catch is you can’t simply use the dummy call to guarantee that the next call to the service will work:
public void SomeMethode()
{
if (ConnectionIsOpen())
{
m_Service.Dummy();
// Connection is lost here
m_Service.SomeMethode();
}
}
To get around this problem, we implemented a system that automatically re-executes any failed service calls which generate a channel fault after the connection has been restored.

The best and asured way to confirm the Communication state is Open or not is to call the Faulted event like below :
proxyInstance.InnerChannel.Faulted -= new EventHandler(ProxyChannelFaulted);
But this works only with those bindings that support ReliableMessaging like WsHttpBinding.
For detail refer the link : WCF Proxy Client taking time to create, any cache or singleton solution for it
Thanks,
Jai Kumar

The fact that you are getting completely different results on windows 7 is not surprising. Microsoft completely re-engineered the TCP stack with windows vista, so the functionality is quite different from xp in the core networking functionality.
The first thing that I would do is use wireshark to see what is actually going across the wire. See if your TCP connection actually terminates when you pull the plug. Windows might be doing some kind of connection persistence / buffering in case the connection comes back quickly.

WCF, BasicHttpBinding: Stop new connections but allow existing connections to continue

.NET 3.5, VS2008, WCF service using BasicHttpBinding
I have a WCF service hosted in a Windows service. When the Windows service shuts down, due to upgrades, scheduled maintenance, etc, I need to gracefully shut down my WCF service. The WCF service has methods that can take up to several seconds to complete, and typical volume is 2-5 method calls per second. I need to shut down the WCF service in a way that allows any previously call methods to complete, while denying any new calls. In this manner, I can reach a quiet state in ~ 5-10 seconds and then complete the shutdown cycle of my Windows service.
Calling ServiceHost.Close seems like the right approach, but it closes client connections right away, without waiting for any methods in progress to complete. My WCF service completes its method, but there is no one to send the response to, because the client has already been disconnected. This is the solution suggested by this question.
Here is the sequence of events:
Client calls method on service, using the VS generated proxy class
Service begins execution of service method
Service receives a request to shut down
Service calls ServiceHost.Close (or BeginClose)
Client is disconnected, and receives a System.ServiceModel.CommunicationException
Service completes service method.
Eventually service detects it has no more work to do (through application logic) and terminates.
What I need is for the client connections to be kept open so the clients know that their service methods completed sucessfully. Right now they just get a closed connection and don't know if the service method completed successfully or not. Prior to using WCF, I was using sockets and was able to do this by controlling the Socket directly. (ie stop the Accept loop while still doing Receive and Send)
It is important that the host HTTP port is closed so that the upstream firewall can direct traffic to another host system, but existing connections are left open to allow the existing method calls to complete.
Is there a way to accomplish this in WCF?
Things I have tried:
ServiceHost.Close() - closes clients right away
ServiceHost.ChannelDispatchers - call Listener.Close() on each - doesn't seem to do anything
ServiceHost.ChannelDispatchers - call CloseInput() on each - closes clients right away
Override ServiceHost.OnClosing() - lets me delay the Close until I decide it is ok to close, but new connections are allowed during this time
Remove the endpoint using the technique described here. This wipes out everything.
Running a network sniffer to observe ServiceHost.Close(). The host just closes the connection, no response is sent.
Thanks
Edit: Unfortunately I cannot implement an application-level advisory response that the system is shutting down, because the clients in the field are already deployed. (I only control the service, not the clients)
Edit: I used the Redgate Reflector to look at Microsoft's implementation of ServiceHost.Close. Unfortunately, it calls some internal helper classes that my code can't access.
Edit: I haven't found the complete solution I was looking for, but Benjamin's suggestion to use the IMessageDispatchInspector to reject requests prior to entering the service method came closest.

Guessing:
Have you tried to grab the binding at runtime (from the endpoints), cast it to BasicHttpBinding and (re)define the properties there?
Best guesses from me:
OpenTimeout
MaxReceivedMessageSize
ReaderQuotas
Those can be set at runtime according to the documentation and seem to allow the desired behaviour (blocking new clients). This wouldn't help with the "upstream firewall/load balancer needs to reroute" part though.
Last guess: Can you (the documention says yes, but I'm not sure what the consequences are) redefine the address of the endpoint(s) to a localhost address on demand?
This might work as a "Port close" for the firewall host as well, if it doesn't kill of all clients anyway..
Edit: While playing with the suggestions above and a limited test I started playing with a message inspector/behavior combination that looks promising for now:
public class WCFFilter : IServiceBehavior, IDispatchMessageInspector {
private readonly object blockLock = new object();
private bool blockCalls = false;
public bool BlockRequests {
get {
lock (blockLock) {
return blockCalls;
}
}
set {
lock (blockLock) {
blockCalls = !blockCalls;
}
}
}
public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
}
public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) {
}
public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) {
foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) {
endpointDispatcher.DispatchRuntime.MessageInspectors.Add(this);
}
}
}
public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) {
lock (blockLock) {
if (blockCalls)
request.Close();
}
return null;
}
public void BeforeSendReply(ref Message reply, object correlationState) {
}
}
Forget about the crappy lock usage etc., but using this with a very simple WCF test (returning a random number with a Thread.Sleep inside) like this:
var sh = new ServiceHost(new WCFTestService(), baseAdresses);
var filter = new WCFFilter();
sh.Description.Behaviors.Add(filter);
and later flipping the BlockRequests property I get the following behavior (again: This is of course a very, very simplified example, but I hope it might work for you anyway):
// I spawn 3 threads
Requesting a number..
Requesting a number..
Requesting a number..
// Server side log for one incoming request
Incoming request for a number.
// Main loop flips the "block everything" bool
Blocking access from here on.
// 3 more clients after that, for good measure
Requesting a number..
Requesting a number..
Requesting a number..
// First request (with server side log, see above) completes sucessfully
Received 1569129641
// All other messages never made it to the server yet and die with a fault
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request before the block.
Error in client request before the block.

Is there an api for the upstream firewall? The way we do this in our application is to stop new requests coming in at the load balancer level, and then when all of the requests have finished processing we can restart the servers and services.

My suggestion is to set an EventHandler when your service goes into a "stopping state", use the OnStop method. Set the EventHandler indicating that your service is going into a stopping state.
Your normal service loop should check if this event is set, if it is, return a "Service is stopping message" to the calling client, and do not allow it to enter your normal routine.
While you still have active processes running, let it finish, before the OnStop method moves on to killing the WCF host (ServiceHost.Close).
Another way is to keep track of the active calls by implementing your own reference counter. you will then know when you can stop the Service Host, once the reference counter hits zero, and by implementing the above check for when the stop event has been initiated.
Hope this helps.

I haven't implemented this myself, so YMMV, but I believe what you're looking to do is pause the service prior to fully stopping it. Pausing can be used to refuse new connections while completing existing requests.
In .NET it appears the approach to pausing the service is to use the ServiceController.

Does this WCF Service authenticate the user in any way?
Do you have any "Handshake" method?

I think you might need to write your own implementation with a helper class that keeps track of all running requests, then when a shutdown is requested, you can find out if anything is still running, delay shutdown based on that... (using a timer maybe?)
Not sure about blocking further incoming requests... you should have a global variable that tells your application whether a shutdown was requested and so you could deny further requests ...
Hope this may help you.

Maybe you should set the
ServiceBehaviorAttribute and the OperationBehavior attribute. Check this on MSDN

In addition to the answer from Matthew Steeples.
Most serious load balancers like a F5 etc. have a mechanism to identify if a node is alive. In your case it seems to check whether a certain port is open. But alternative ways can be configured easily.
So you could expose e.g. two services: the real service that serves requests, and a monitoring "heart beat"-like service. When transitioning into maintenance mode, you could first take the monitoring service offline which will take the load away from the node and only shutdown the real service after all requests finished processing. Sounds a bit weird but might help in your scenario...

How to check the availability of a net.tcp WCF service

My WCF server needs to go up and down on a regular basis, the client sometimes uses the server, but if it is down the client just ignore it.
So each time I need to use the server services I check the connection state and if it's not open I open it.
The problem is that if I attempt to open while the server is down there is a delay which hits performance.
My question is, is there a way to do some kind of myClient.CanOpen()? so I'd know if there is any point to open the connection to the server.

There is an implementation of WS-Discovery that would allow you to listen for up/down announcements for your service. This is also a very convenient form of service address resolution because it utilizes UDP multicast messages to find the service, rather than configuring one set address on the client.
WS-Discovery for WCF
There's also an implementation done by a Microsoft employee:
WS-Discovery Sample Implementation
.NET 4.0 will include this natively. You can read about .NET 4.0's implementation on Jesus Rodriguez's blog. It has a great chart that details the ad-hoc communication that goes on in WS-Disco Using WS-Discovery in WCF 4.0
Another thing you might consider, especially if your messages are largely one-way, is a protocol that works natively disconnected, like MSMQ. I don't know what your design for your application looks like, but MSMQ would allow a client to send a message regardless of the state of the service and the service will get it when it comes back up. This way your client doesn't have to block quite so much trying to get confirmation that a service is up before communicating... it'll just fire and forget.
Hope this helps.

If you are doing a synchronous call expecting a server timeout in an application with a user interface, you should be doing it in another thread. I doubt that the performance hit is due to exception overhead.
Is your performance penalty in CPU load, gui availability or wall clock time?
You could investigate to see if you can create a custom binding on TCP, but with faster timeout.
I assume you know that "IsOneWay=true" is faster than request->response in your case because you wouldn't be expecting a response anyway, but then you are not getting confirmation or return values.
You could also implement a two-way communication that is not request->response.

If you were in a local network it might be possible to broadcast a signal to say that a new server is up. The client would need to listen for the broadcast signal and respond accordingly.

Here's what I'm using and it works like a charm. And btw, the ServiceController class lives in namespace 'System.ServiceProcess'.
try
{
ServiceController sc = new ServiceController("Service Name", "Computer's IP Address");
Console.WriteLine("The service status is currently set to {0}",
sc.Status.ToString());
if ((sc.Status.Equals(ServiceControllerStatus.Stopped)) ||
(sc.Status.Equals(ServiceControllerStatus.StopPending)))
{
Console.WriteLine("Service is Stopped, Ending the application...");
Console.Read();
EndApplication();
}
else
{
Console.WriteLine("Service is Started...");
}
}
catch (Exception)
{
Console.WriteLine("Error Occurred trying to access the Server service...");
Console.Read();
EndApplication();
}

I don't think it's possible doing a server side call to your Client to inform him that you the service has been started ... Best method i can see is having a client method figuring out where or not the service is open and in good condition. Unless I am missing some functionality of WCF ...
There is a good blogpost WCF: Availability of the WCF services if you are interested in a read.