Description about the problem.
Currently I am developing server - multiple clients (over 3000 clients) application in csharp. Server and clients exchange messages. The problem is that server get crashed if any of the client crashes for the internal reason. The internal reason can include all sort of specific problem of client computer. For example, they can be the lost internet connection, the computer crash, power failure, etc.
As the server functions to deliver the real time information, if the server crashes, then entire operation stops. This is nightmare for server because we can not predict which client will crash.
The main code causing the server crashes is below:
public bool Send(byte[] buffer)
{
if (m_Socket == null)
{
return (false);
}
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
return (false);
}
I think there were few answers to the similar problem from my research. However often the answers point out the use of Keep alive packets.
I think many answer in this website indicates that detecting the half open connection (dropped connection) is almost impossible without using keep alive packet from client side.
In our application, we do not prefer to use Keep alive packet or message because the server and client communicate in milliseconds. Receiving keep alive packet or message from over 3000 clients every seconds does not seems friendly for resource management point of view for server. So we prefer to have rather a good error management. If the client computer is crashed, then we just want to ignore. I am not sure whether this can make the running server unstable ?
Anyway, I came up with few ideas with below options. It was the suggestions from other coders most of time on the net. So the question is Which option might be the best to safely catch the Socket.Send() failure error and not disturbing the operation of the server?
Using simple try catch statement. Will this offer the 100% secure fail safe operation for this problem ? From my experience, sometimes, it works but sometimes it does not work.
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
Use of while(true) loop. Someone mentioned that while(true) loop might be the safe way of doing this. However, I am not sure how effective this solution comparing to simple try catch statement.
Perform Socket.Send() function inside other thread rather than main thread. The idea is that if Socket.Send() functions fails then the thread used for Socket.send() function will terminate on its own without disturbing the main thread for the application. However, I do not have sufficient knowledge how each thread suppose to interact together. So I can not tell whether this method is better than above two methods.
Can you please suggest any idea or solution for this specific problem? I think that without using Keep alive packet, maintaining the sever operation without crash is quite challenging problem.
Related
Basically the title... I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value. This is the starting code:
static void Main(string[] args){
Console.WriteLine("Client running!");
NamedPipeClientStream npc = new NamedPipeClientStream("somename");
npc.Connect();
// npc.WriteTimeout = 1000; does not work, says it is not supported for this stream
byte[] message = Encoding.UTF8.GetBytes("Message");
npc.Write(message);
int response = npc.ReadByte();
Console.WriteLine("response; "+response);
}
I've implemented a small echo message from the NamedPipeServerStream on every read. I imagine I could add some async timeout to check if npc.ReadByte(); did return a value in lets say 200ms. Similar to how TCP packets are ACKed.
Is there a better way of inspecting if namedPipeClientStream.Write() was successful?
I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value
The only way to know for sure that the data you sent was received and successfully processed by the client at the remote endpoint, is for your own application protocol to include such acknowledgements.
As a general rule, you can assume that if your send operations are completing successfully, the connection remains viable and the remote endpoint is getting the data. If something happens to the connection, you'll eventually get an error while sending data.
However, this assumption only goes so far. Network I/O is buffered, usually at several levels. Any of your send operations almost certainly involve doing nothing more than placing the data in a local buffer for the network layer. The method call for the operation will return as soon as the data has been buffered, without regard for whether the remote endpoint has received it (and in fact, almost never will have by the time your call returns).
So if and when such a call throws an exception or otherwise reports an error, it's entirely possible that some of the previously sent data has also been lost in transit.
How best to address this possibility depends on what you're trying to do. But in general, you should not worry about it at all. It will typically not matter if a specific transmission has been received. As long as you can continue transmitting without error, the connection is fine, and asking for acknowledgement is just unnecessary overhead.
If you want to handle the case where an error occurs, invalidating the connection, forcing you to retry, and you want to make the broader operation resumable (e.g. you're streaming some data to the remote endpoint and want to ensure all of the data has been received, without having to resend data that has already been received), then you should build into your application protocol the ability to resume, where on reconnecting the remote endpoint reports the number of bytes it's received so far, or the most recent message ID, or whatever it is your application protocol would need to understand where it needs to start sending again.
See also this very closely-related question (arguably maybe even an actual duplicate…though it doesn't mention named pipes specifically, pretty much all network I/O will involve similar issues):
Does TcpClient write method guarantees the data are delivered to server?
There's a good answer there, as well as links to even more useful Q&A in that answer.
I am currently working with a server application we have designed to communicate with a Xamarin mobile app. We are using an old messaging library that makes a connection with a TcpClient and keeps the connection open (with a heartbeat message every 3 seconds). We added SSL to the library by wrapping the TcpClient stream with an SslStream. We have run the server application on Windows and it works well, but our ultimate target is Mono on a BeagleBoneBlack.
However, when we close the stream and the client on the mobile app side and then attempt to re-initiate a new connection, the SslStream.AuthenticateAsServer(...) will not complete on the server. However, if I completely close the mobile app, the server will throw an exception. At that point, I re-open the app, and can reconnect without any issue.
So it seems something low level is not being closed on either the app or the server side. What is odd is that I run the exact same code on both when the server is running on windows and I don't have an issue.
Here is my code that closes/disposes the stream
public async Task Disconnect()
{
if (!UseAsync)
{
semaphore.Wait();
}
try
{
if (UseSSL)
{
SslStream?.Close();
}
Client?.GetStream()?.Close();
Client?.Close();
}
catch (Exception ex) // Assuming we had an exception from trying to close the sslstream
{
logger.Error(ex, "Did not close/dispose correctly: {0}", ex.ToString());
}
finally
{
SslStream = null;
Client = null;
if (!UseAsync)
{
semaphore.Release();
}
}
}
Edit: It shouldn't be significant since the issue seems to lie at the server somewhere, and the client and server ssl code is almost identical, but in case someone asks, here is the client disconnect code
public async Task Disconnect()
{
try
{
if (UseSSL)
{
_sslStream?.Close();
}
Client?.GetStream()?.Close();
Client?.Close();
}
catch // Assuming we had an exception from trying to close the sslstream
{
// Ignore exceptions since we've already closed them
}
finally
{
_sslStream = null;
Client = null;
}
}
Edit 2
It should also be noted that I've found at least one bug report that looks like it's the same issue I'm dealing with. It doesn't appear from this bug report that it has ever been resolved, but I found other reports that seemed to reflect a similar issue in the mono framework and it was resolved. Additionally, I have added code to send some "dummy" data from the client after the connect ant it seems to have no affect.
Edit 3: I ultimately receive this exception on the client side
System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x0003a] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/RecordProtocol.cs:430
at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslClientStream.cs:256
at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00360] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslClientStream.cs:533
Well... I almost feel silly answering this, but I feel others could end up in my situation out of equal ignorance, so hopefully this helps someone out with a similar architecture.
What happened is that I created a self signed certificate and had it in my project folder, so the same certificate was being used on all of my "server" instances. What was happening is that, due to some internal caching of the SSL session cache, after connecting to one device, it would cache its information and then re-use that information when connecting the next time. The result would be that, since the self-signed certificate used the same host name for all devices, once it tried to reconnect to device B after being connected to device A, it would attempt to re-use device A's cached information (as far as I can tell) The reason this didn't initially make sense is that I could connect and disconnect to multiple different Windows servers over and over again, but only when connecting to Mono servers did I see this issue. As a result, it still seems to be a bug in either Windows or Mono's SSLStream implementation (since, in a perfect world, they are identical), but unfortunately I don't have the time to dig through the de-compiled source code to find it. And frankly, it probably doesn't matter because what I was doing was breaking the whole notion of an SSL connection anyways.
Ultimately, I created a function to programmatically generate a unique certificate for each device using Mono.Security and then provide a mechanism to provide the client with the unique hostname (even though the client is connecting directly to an IP address).
Hello and thanks for your help.
This time I would like to ask about TcpClient.
I have a server program and I am writing a client program.
This client uses TcpClient. It starts by creating a new client
clientSocket=new TcpClient();
(By the way, can this cause exceptions? just in case I put it inside a try-catch but I am not sure if that is really necessary)
Anyway, later I enter a loop and inside this loop I connect to the server
clientSocket.Connect("xx.xx.xx.xx",port);
Then I create a NetworkStream with
clientStream=clientSocket.GetStream();
and then start waiting for data from the server through Read. I know this is blocking so I also set a ReadTimeOut (say 1 second)
Anyway, so far so good.
Later if I don't receive anything from the server, I attempt to send something to it. If this keeps happening for say 3 times I want to close the connection and reconnect to the server again
(notice that a whole different problem is when the server somehow is down, cause that causes other kinds of errors in the client-perhaps I will ask about that later)
So, what do I do?
if(clientSocket.Connected)
{
Console.WriteLine("Closing the socket");
clientSocket.Close();
}
I close the socket.
The loop is finished so I go again to the beginning and try to connect to the server.
clientSocket.Connect("xx.xx.xx.xx",port);
However this causes an error(an unhandled exception actually) "Can not access a disposed object"
So my question is How can I close and reconnect to the server again??
Thanks again for any help
A TcpClient instance can only be used to connect once. You can simply instantiate a new TcpClient, rather than trying to re-open a closed one.
As explained in the other answer, a TcpClient object can only be connected once. If you want to reconnect to the server, you have to create a new TcpClient object and call Connect() again.
That said, you have a number of apparent misconceptions in your question:
First and most important, you should not use ReceiveTimeout if you have any intention whatsoever of trying to use the TcpClient object again, e.g. to send some data to the server. Once the timeout period has expired, the underlying socket is no longer usable.If you want to periodically send data to the server when the server hasn't sent data to you, you should use asynchronous I/O (which you should do anyway, in spite of the learning curve) and use a regular timer object to keep track of how long it's been since you received data from the server.
The TcpClient constructor certainly can throw an exception. At the very least, any attempt to new a reference type object could throw OutOfMemoryException, and in the case of TcpClient, it ultimately tries to create a native socket handle, which could also fail.While all I/O objects and methods can throw exceptions, you should only ever catch exceptions that you have a way to handle gracefully. So before you add a try/catch block to your code, decide what it is you want to do in the case of an exception that will ensure that your code doesn't corrupt any data and continues to operate correctly. It is generally not possible to gracefully handle OutOfMemoryException (and impractical to protect all uses of new in any case), but you certainly can catch SocketException, which could be thrown by the constructor. If that exception is thrown, you should immediately abandon the attempt to create and use TcpClient, and report the error the user so that they can attempt to correct whatever problem prevented the socket's creation.
If your server is expected to be sending you data, and you don't receive it, then closing the connection and retrying is unlikely to improve the situation. That will only cause additional load on the server, making it even more likely it will fail to respond. Likewise sending the same data over and over. You should your request once, wait as long as is practical for a response from the server, and if you get no response within the desired time, report the error to the user and let them decide what to do next.Note that in this case, you could use the ReceiveTimeout property, because all you're going to do if you don't get a response in time is abandon the connection, which is fine.
Very simple:
client.Close();
client = new TcpClient();
client.Connect(host, port);
I am using multithreaded server to handle client communication, I don't know how many clients can this server handle. If number of clients increase will it be able to handle them? I am using it on core2duo processor.
Will starting server on different port solve the problem, if i redirect half of the clients to new server with another port?
Here is my code for server
Public void ListenForClients()
{
this.tcpListener.Start();
while (true)
{
try
{
TcpClient client = this.tcpListener.AcceptTcpClient();
NetworkStream clientStream = client.GetStream(); //create networkstream for connected client
Console.WriteLine(((IPEndPoint)client.Client.RemoteEndPoint).Address.ToString());//client ipaddress
Console.WriteLine("connecting..");
Thread clientThread = new Thread(new ParameterizedThreadStart(updatedb));
clientThread.Start(client);
}
catch (Exception ex)
{
Console.WriteLine("exception" + ex.ToString());
Console.ReadLine();
}
}
}
I suggest you read this blog post and similar posts by the author. He explains there in great detail how multi threading is used by IIS / ASP.NET.
What you must remember, is that even if You can create more threads (by running anothre instance of your application, as you suggested, for example), it doesnt mean your application will be more responsive / return the expected answers to the clients faster, as there can be only as much running threads at any given moment as the number of CPUS your server has.
I dont think you need to write your code in an expectation of collapsing, but to work in a direction of more of an async processing, as the ISS server (which is practically doing the same - servicing TCP connections) is doing. Theres no reason not to use the thread pool provided by .NET, let it handle the real number of co-existing threads at every given moment, and let the other requests queue untill a thread becomes availibale.
Without actually trying it, everything else is a guess. As Hans Passant says, your strategy of creating a new thread for every request will not scale well. It may seem ok at first, but you should see performance fall off badly over a few hundred simultaneous users. In addition, it looks like the work will be communicating with a database (updatedb), so these threads you've created will just be sending data to an external process and waiting for a reply? That's the worst use of a thread. See if you can use asynchronous sql updates.
C# 2008
I am using this code to test for an internet connection. As my application will have to login to a web server. However, if the user internet connection was to fail or cable pulled out. I will have to notify the user.
// Ping www.google.com to check if the user has a internet connection.
public bool PingTest()
{
Ping ping = new Ping();
PingReply pingStatus = ping.Send(IPAddress.Parse("208.69.34.231"));
if (pingStatus.Status == IPStatus.Success)
{
return true;
}
else
{
return false;
}
}
The only way I think I can test for an Internet connection is to ping www.google.com. So I have a used a server timer which I have set for 500 milliseconds and in the lapsed event it will call this ping function. If the ping returns false. Then my app will take appropriate action.
Do you think using google as a way to test an Internet connect is a good thing. If google was to fail, then my app would not function. Is polling 1/2 second to much or too little? Just wondering about my whole idea if it is good or not?
Many thanks,
Why ping Google? The only server you really care about is the web server you want to talk to, right? So ping that instead. That way, you can notify the user if anything is wrong between them and the web server - that's the only thing that matters.
If something goes wrong with the internet which means that the user can only do anything within their own town, you may not be able to reach Google but if you can still reach the web server you're interested in, why report a problem? On the other hand, if the web server goes down but Google doesn't, why would you not want to tell the user that there's a problem?
Likewise, why use a ping? Why not send an HTTP request to a known URL on the web server? After all, if the network is up but the web server process itself is down, surely that's just as bad as the user's network going down.
EDIT: I hadn't noticed in the question that you were repeating this every half second. That's really not pleasant for anyone. Surely once a minute is often enough, isn't it?
Check out this duplicate question which has a good answer that doesn't require ping:
C# - How do I check for a network connection
You could subscribe to the System.Net.NetworkInformation.NetworkChange.NetworkAvailabilityChanged event and only then when it indicates that the network is down do a ping/web request to the server to see if it's available.
I might be missing something but, Why do you need to test that the server is up and the web service running before you need to use it? If you call a method in your web services and you don't get a timely response, then take what ever action you see fit or need?
It just seems over engineering of the design.
[edit]
Thanks for the clarification robUK. You should listen to NET Rocks podcast episode 'Udi Dahan Scales Web Applications!' (date 12/08/2008). They discuss adding a GUID to a request and wait until you get the same GUID in the response. Maybe you could fire it off every second for say 10 attempts and if you don't get a response back (with the GUID attached) after 10 attempts, then take the required action?
Good luck with it..
I have an app that needs to do something similar. We have a bit of a different architecture in place, namely a pub/sub message bus. For those applications interested in hearing "heartbeats" from the various services, we allow them to register their interest with the bus. Every few second or so, each service will publish a health state message that consumers will then receive. If the consumers do not receive a health state message for a certain amount of time, they can take appropriate action until they hear it again.
This setup is much more scalable than polling the server or even worse pinging it. Imagine if you had 100 or 1000 clients all polling or pinging your sever every few seconds. Bad.
you can simply test like this
[DllImport("wininet.dll")]
private extern static bool InternetGetConnectedState(out int conn, int val);
C# full source code
http://net-informations.com/csprj/communications/internet-connection.htm
It'd suggest to put more than 0.5s, because sometimes your lataency can get higher. I sometimes have even 3.5s on DSL.
Better to see if you are on the network than to bother a server pinging it. See Detect Internet V. local lan connection for example. If you are on Windows 7 / Server 2008 R2 you can get an event when connectivity changes, which is going to be much happier for everyone than constant pinging and polling.
Even though I agree with Jon that pinging your webserver might be the best idea I want to throw in another solution that might be used stand alone or together with an initial ping to the service.
These is a built in event on the NetworkChange-class called NetworkAvailabilityChanged that you can use to get the overall status from Windows whether you are online or not. Just add a listener to it and then you will be notified when it changes.
private void MyForm_Load(object sender, EventArgs e)
{
NetworkChange.NetworkAvailabilityChanged += new NetworkAvailabilityChangedEventHandler(NetworkChange_NetworkAvailabilityChanged);
}
private void NetworkChange_NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
{
if (e.IsAvailable)
{
WriteLog("Network is available again, updating items");
timer1_Tick(sender, EventArgs.Empty);
return;
}
WriteLog("Network isn't available at the moment");
}
It's available from .Net 2.0 and onward. More info can be found on MSDN.