SslStream on Mono not fully closing - c#

I am currently working with a server application we have designed to communicate with a Xamarin mobile app. We are using an old messaging library that makes a connection with a TcpClient and keeps the connection open (with a heartbeat message every 3 seconds). We added SSL to the library by wrapping the TcpClient stream with an SslStream. We have run the server application on Windows and it works well, but our ultimate target is Mono on a BeagleBoneBlack.
However, when we close the stream and the client on the mobile app side and then attempt to re-initiate a new connection, the SslStream.AuthenticateAsServer(...) will not complete on the server. However, if I completely close the mobile app, the server will throw an exception. At that point, I re-open the app, and can reconnect without any issue.
So it seems something low level is not being closed on either the app or the server side. What is odd is that I run the exact same code on both when the server is running on windows and I don't have an issue.
Here is my code that closes/disposes the stream
public async Task Disconnect()
{
if (!UseAsync)
{
semaphore.Wait();
}
try
{
if (UseSSL)
{
SslStream?.Close();
}
Client?.GetStream()?.Close();
Client?.Close();
}
catch (Exception ex) // Assuming we had an exception from trying to close the sslstream
{
logger.Error(ex, "Did not close/dispose correctly: {0}", ex.ToString());
}
finally
{
SslStream = null;
Client = null;
if (!UseAsync)
{
semaphore.Release();
}
}
}
Edit: It shouldn't be significant since the issue seems to lie at the server somewhere, and the client and server ssl code is almost identical, but in case someone asks, here is the client disconnect code
public async Task Disconnect()
{
try
{
if (UseSSL)
{
_sslStream?.Close();
}
Client?.GetStream()?.Close();
Client?.Close();
}
catch // Assuming we had an exception from trying to close the sslstream
{
// Ignore exceptions since we've already closed them
}
finally
{
_sslStream = null;
Client = null;
}
}
Edit 2
It should also be noted that I've found at least one bug report that looks like it's the same issue I'm dealing with. It doesn't appear from this bug report that it has ever been resolved, but I found other reports that seemed to reflect a similar issue in the mono framework and it was resolved. Additionally, I have added code to send some "dummy" data from the client after the connect ant it seems to have no affect.
Edit 3: I ultimately receive this exception on the client side
System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x0003a] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/RecordProtocol.cs:430
at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslClientStream.cs:256
at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00360] in /Users/builder/data/lanes/3511/501e63ce/source/mono/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslClientStream.cs:533

Well... I almost feel silly answering this, but I feel others could end up in my situation out of equal ignorance, so hopefully this helps someone out with a similar architecture.
What happened is that I created a self signed certificate and had it in my project folder, so the same certificate was being used on all of my "server" instances. What was happening is that, due to some internal caching of the SSL session cache, after connecting to one device, it would cache its information and then re-use that information when connecting the next time. The result would be that, since the self-signed certificate used the same host name for all devices, once it tried to reconnect to device B after being connected to device A, it would attempt to re-use device A's cached information (as far as I can tell) The reason this didn't initially make sense is that I could connect and disconnect to multiple different Windows servers over and over again, but only when connecting to Mono servers did I see this issue. As a result, it still seems to be a bug in either Windows or Mono's SSLStream implementation (since, in a perfect world, they are identical), but unfortunately I don't have the time to dig through the de-compiled source code to find it. And frankly, it probably doesn't matter because what I was doing was breaking the whole notion of an SSL connection anyways.
Ultimately, I created a function to programmatically generate a unique certificate for each device using Mono.Security and then provide a mechanism to provide the client with the unique hostname (even though the client is connecting directly to an IP address).

Related

How to trap when server forcebly closes HttpClient

One is encouraged to re-use HttpClient rather than making a new one for each api call.
https://learn.microsoft.com/en-us/aspnet/web-api/overview/advanced/calling-a-web-api-from-a-net-client
My client program has a single connection that remains open.
However, if somebody stops IIS then the connection can't remain open.
In this case I get an exception that kills my client program even though I have a catch for it. (see code)
Is there some way to make the exception handler handle the exception without crashing the program?
Alternatively, is there some property of HttpClient that indicates that the server has closed it?
try
{
return await m_Client.GetAsync(AppVars.Instance.ServerAddress + method);
}
catch (Exception ex)
{
// getting 'An existing connection was forcibly closed by the remote host' kills the
// application
// getting 'No connection could be made because the target machine actively refused it'
// does not kill the application - i.e. catch behaves as expected
}
I have discovered that this is caused by Visual Studio. I am debugging both the client and the server from one solution (multiple startup projects). When I close IIS Express, Visual Studio kills the client next time it tries to use the connection. If I launch a separate instance of Visual Studio and debug the client from there then the client does not get killed when I try to re-use the closed connection, i.e. the exception handling works as expected. (so I can make a new connection and retry)
Hopefully this will help anyone who experiences the same issue.

How to Stop Server Crash when Client Crashed internally (TCP/IP Socket)

Description about the problem.
Currently I am developing server - multiple clients (over 3000 clients) application in csharp. Server and clients exchange messages. The problem is that server get crashed if any of the client crashes for the internal reason. The internal reason can include all sort of specific problem of client computer. For example, they can be the lost internet connection, the computer crash, power failure, etc.
As the server functions to deliver the real time information, if the server crashes, then entire operation stops. This is nightmare for server because we can not predict which client will crash.
The main code causing the server crashes is below:
public bool Send(byte[] buffer)
{
if (m_Socket == null)
{
return (false);
}
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
return (false);
}
I think there were few answers to the similar problem from my research. However often the answers point out the use of Keep alive packets.
I think many answer in this website indicates that detecting the half open connection (dropped connection) is almost impossible without using keep alive packet from client side.
In our application, we do not prefer to use Keep alive packet or message because the server and client communicate in milliseconds. Receiving keep alive packet or message from over 3000 clients every seconds does not seems friendly for resource management point of view for server. So we prefer to have rather a good error management. If the client computer is crashed, then we just want to ignore. I am not sure whether this can make the running server unstable ?
Anyway, I came up with few ideas with below options. It was the suggestions from other coders most of time on the net. So the question is Which option might be the best to safely catch the Socket.Send() failure error and not disturbing the operation of the server?
Using simple try catch statement. Will this offer the 100% secure fail safe operation for this problem ? From my experience, sometimes, it works but sometimes it does not work.
try
{
mSocket.Send(buffer);
return (true);
}
catch(System.Net.Sockets.SocketException ex)
{
mSocket = null;
return (false);
}
Use of while(true) loop. Someone mentioned that while(true) loop might be the safe way of doing this. However, I am not sure how effective this solution comparing to simple try catch statement.
Perform Socket.Send() function inside other thread rather than main thread. The idea is that if Socket.Send() functions fails then the thread used for Socket.send() function will terminate on its own without disturbing the main thread for the application. However, I do not have sufficient knowledge how each thread suppose to interact together. So I can not tell whether this method is better than above two methods.
Can you please suggest any idea or solution for this specific problem? I think that without using Keep alive packet, maintaining the sever operation without crash is quite challenging problem.

Wince Socket exception on asynchronous HTTP request

I am writing a WinCE app in C# that makes an HTTP POST to an APACHE server residing on my network. Because of some network issues (I am guessing), I get the following exception in the managed code
System.Net.Sockets.SocketException occurred
Message="A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
ErrorCode=10060
NativeErrorCode=10060
StackTrace:
at System.Net.Sockets.Socket.ConnectNoCheck(EndPoint remoteEP)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Net.Connection.doConnect(IPEndPoint ep)
at System.Net.Connection.connect(Object ignored)
at System.Threading.ThreadPool.WorkItem.doWork(Object o)
at System.Threading.Timer.ring()
This exception isn't always thrown, but when it is thrown, my app fails to connect to the server AT ALL. Repeated connection attempts don't help in reconnecting either. The only thing that helps is closing and re-deploying the app.
I can't catch the exception because its inside of managed code. Is there any way to circumvent this and close all socket connections to my server and re-initialize them? Is there something I am doing wrong?
The exception message looks a bit misleading ("connection attempt failed because the connected party") but I think it means your hardware is communicating with the server, but the server is not accepting the connection on the TCP level.
A problem I could think of is "hanging" connections, causing the server to reach the maximum number of concurrent connections and to stop accepting new ones.
Although it's just a guess, you might want to check the apache log if you can to see if you can find out if the server reports anything, and perhaps try restarting apache as soon as the problem occurs again. If that helps, you still need to find the cause of course.

Getting error reason code 2059 on MQ client (C#) when reconnecting to QueueManager after awhile

I can't reconnect to MQQueueManager after a while as an exception (reason code 2059 - MQRC_Q_MGR_NOT_AVAILABLE) is thrown when I'm constructing new object of MQQueueManager. My client app is written in .NET/C# and I'm running it on Win2003.
However I can connect to QM after I have restarted my client app. This would indicate that some state is incorrect in QM libraries? How can I reset the state in code so that I could reconnect to QM? Is there a way to reset/disconnect all active TCP connections to QM from client app code?
My connection code:
Hashtable properties = new Hashtable();
properties.Add( MQC.HOST_NAME_PROPERTY, Host );
properties.Add( MQC.PORT_PROPERTY, Port );
properties.Add( MQC.USER_ID_PROPERTY, UserId );
properties.Add( MQC.PASSWORD_PROPERTY, Password );
properties.Add( MQC.CHANNEL_PROPERTY, ChannelName );
properties.Add( MQC.TRANSPORT_PROPERTY, TransportType );
// Following line throws an exception randomly
MQQueueManager queueManager = new MQQueueManager( qmName, properties );
Stack trace:
Source: amqmdnet
CompletionCode: 2
ReasonCode: 2059
Reason: 2059
Stack Trace:
at IBM.WMQ.MQBase.throwNewMQException()
at IBM.WMQ.MQQueueManager.Connect(String queueManagerName)
at IBM.WMQ.MQQueueManager..ctor(String qmName, Hashtable properties)
at WebSphereMQOutboundAdapter.WebSphereMQOutbound.ConnectToWebSphereMQ()
Connections are per-thread so if you are attempting to create a new connection while the previous QMgr object is still instantiated, you would get this. If you close the previous connection and destroy the object before creating a new object you should be OK. Since queues and other WMQ objects depend on a connection handle these will also need to be destroyed and then reinstantiated after the new connection is made.
There are of course a few other explanations for this behavior but these are much less likely. For example, it is possible that a channel exit or (in WMQ v7) configuration could be limiting the number of simultaneous connections from a given IP address. When a connection is severed rather than closed, the channel agent holding the connection on the QMgr side has to time out before the QMgr sees the connection as closed. If connection limiting is in place, these "ghost" connections reduce the available pool. But as I said, this is far less common than programs not cleaning up old objects prior to a reconnect attempt.
There is also the possibility that this is a bug. To reduce that possibility, and for a variety of other reasons such as WMQ v6 going end of life next year, I'd recommend use of WMQ v7.0.1.2 for this project, at both the client and server side. In general, you can use v7.0.1.2 client with a v6.0.x server as long as you stick to v6 functionality. Among other things, .Net code is better integrated in v7 and the Cat-3 SupportPacs are now included in the base install media rather than a separate download.
After some months fighting with this issue and IBM support, the best solution I found is to change the connect/disconnect code in IBM MQ Driver.
Instead of calling manager.Disconnect() and manager.Close() for each GET/PUT, connect once and then reconnect only if you have some exception (like loosing connection).
What I've figure out is that some bug exists in IBM MQ Driver that caches some information for each connect/disconnect. When this buffer is full, the application stops reconnecting.
The driver version (client DLL's) I have this issue is: 7.0.1.6

An existing connection was forcibly closed by the remote host

I am working with a commercial application which is throwing a SocketException with the message,
An existing connection was forcibly closed by the remote host
This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.
Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?
Any comments / ideas are welcome.
... The latest ...
I have some logging from some .NET tracing,
System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0
Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?
One of two possibilities is occurring, and I am not sure which,
The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.
The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.
What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?
This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
The network link between the client and server is going down for some reason
You have triggered a bug in the third-party application that caused it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.
Without more specific information, it's unlikely that anyone here can really help you much.
Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
You need to reboot your computer for these changes to take effect.
This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}
Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:
SocketException: An existing connection was forcibly closed by the
remote host
Simple solution for this common annoying issue:
Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).
Then, add this line to your constructor:
public DBEntities()
: base("name=DBEntities")
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
}
I've got this exception because of circular reference in entity.In entity that look like
public class Catalog
{
public int Id { get; set; }
public int ParentId { get; set; }
public Catalog Parent { get; set; }
public ICollection<Catalog> ChildCatalogs { get; set; }
}
I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.
If Running In A .Net 4.5.2 Service
For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed #willmaz suggestion but got a new error.
In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.
Solution
It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.
I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.
For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:
using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
var buffer = new byte[1024];
int read;
while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await content.WriteAsync(buffer, 0, read);
await content.FlushAsync();
}
}
After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:
using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
await remoteStream.CopyToAsync(content);
}
CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?
It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:
"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."
The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link
In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.
I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.
For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list.
I just had to put in SMTP access request for that app server.
This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):
1. Log into active L.B.
2. Select: Local Traffic > iRules > Data Group List
3. Select the appropriate Data Group
4. Enter the app server's IP address
5. Select: Add
6. Select: Update
7. Sync config changes
Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.
I was trying to connect to the MQTT broker using the GO client,
broker address was given as address + port, or tcp://address:port
Example: ❌
mqtt://test.mosquitto.org
which indicates that you wish to establish an unencrypted connection.
To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.
Example: ✅
mqtts://test.mosquitto.org
In my case, enable the IIS server & then restart and check again.
We are using a SpringBoot service. Our restTemplate code looks like below:
#Bean
public RestTemplate restTemplate(final RestTemplateBuilder builder) {
return builder.requestFactory(() -> {
final ConnectionPool okHttpConnectionPool =
new ConnectionPool(50, 30, TimeUnit.SECONDS);
final OkHttpClient okHttpClient =
new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
// .connectTimeout(30, TimeUnit.SECONDS)
.retryOnConnectionFailure(false).build();
return new OkHttp3ClientHttpRequestFactory(okHttpClient);
}).build();
}
All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.
This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.
This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.

Categories