I have some difficulties resolving an SSL/TLS error while consuming a REST web service.
The REST web service is developed with the Spring Framework, using Spring Boot.
The client is a .NET application, using RestSharp for the REST web service consumption.
Everything worked fine until one client had an error during a file upload.
System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
I was still using the embedded Tomcat v8.5.5 at this time. On my Spring Boot application I had the following errors:
java.lang.NullPointerException: null
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.getSslSupport(NioEndpoint.java:1329) ~[tomcat-embed-core-8.5.5.jar!/:8.5.5]
// Etc.
java.lang.NullPointerException: null
at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:182) ~[tomcat-embed-core-8.5.5.jar!/:8.5.5]
// Etc.
After some researches, I found this topic about similar errors related to the embedded Tomcat v8.5.5. Following their advices, I upgraded the embedded Tomcat to v8.5.13.
Nice, no more NPE from apache Tomcat. So, my .NET application should work... or not.
System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
Still the same error. Since I had no Tomcat error anymore, I changed the logging level of org.apache to DEBUG. And I found some new errors at the DEBUG level:
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.a.tomcat.util.net.SocketWrapperBase : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#3678fe26:org.apache.tomcat.util.net.SecureNioChannel#63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Read from buffer: [0]
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
java.io.EOFException: null
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1242) ~[tomcat-embed-core-8.5.13.jar!/:8.5.13]
// Etc.
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#3678fe26:org.apache.tomcat.util.net.SecureNioChannel#63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Status in: [OPEN_READ], State out: [CLOSED]
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] org.apache.tomcat.util.net.NioEndpoint : Failed to close socket
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) ~[na:1.8.0_91]
// Etc.
2017-05-10 14:58:05.889 DEBUG 928 --- [-nio-443-exec-4] o.apache.tomcat.util.threads.LimitLatch : Counting down[https-jsse-nio-443-exec-4] latch=1
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.a.tomcat.util.net.SocketWrapperBase : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#58361212:org.apache.tomcat.util.net.SecureNioChannel#63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Read from buffer: [0]
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:604) ~[tomcat-embed-core-8.5.13.jar!/:8.5.13]
// Etc.
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.coyote.http11.Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#58361212:org.apache.tomcat.util.net.SecureNioChannel#63c88911:java.nio.channels.SocketChannel[connected local=/... remote=/...]], Status in: [OPEN_READ], State out: [CLOSED]
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] org.apache.tomcat.util.net.NioEndpoint : Failed to close socket
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(Unknown Source) ~[na:1.8.0_91]
// Etc.
2017-05-10 15:00:34.593 DEBUG 928 --- [-nio-443-exec-9] o.apache.tomcat.util.threads.LimitLatch : Counting down[https-jsse-nio-443-exec-9] latch=1
For what I understand, the client seems to successfully connect to the server but proceed to interrupt the connection during file transfer because of an SSL/TLS error, resulting in those ClosedChannelException server-side.
Am I using RestSharp wrongly?
RestRequest request = new RestRequest("path/to/uri", Method.PUT);
request.AddQueryParameter("myParam", myParam.ToString());
request.AddFile("myFile", "path/to/file", "multipart");
IRestResponse<MyResponse> response = myRestClient.Execute<MyResponse>(request);
Spring Boot code for this request:
#RequestMapping(method = RequestMethod.PUT, consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public MyResponse replace(#RequestParam Integer myParam, #RequestParam MultipartFile myFile) {
// Etc.
The PUT method for multipart is not supported by default in Spring but I created my own MultipartResolver to handle it so the problem is not here.
It looks like the problem only happens in one place, and only during file transfer. All other places work fine, but there are still the EOFException and ClosedChannelException server-side, even when it works client-side.
Another obscure thing... it only happens sometimes. Sometimes the upload works, sometimes not.
I am not an SSL expert so I wonder what could do that.
Something not synchronized between the client and the server?
An interruption between the client and the server, resulting the ClosedChannelException?
I am open to all suggestions and will continue to investigate, and hopefully find something...
EDIT 2017-05-11
Precision from comments:
The .NET application is built with the Microsoft Windows .NET Framework (not Mono).
The SSL certificate (server-side) is a complete chain in a PKCS#12 file.
Information retrieved since I posted the question:
The bug only happens if the .NET application is open for a certain amount of time; it will not occur if opened and closed in less than a minute.
Host OS: Windows 10 Pro x64... but System.Environment.OSVersion.VersionString gives Windows NT 6.2.9200.0. According to Wikipedia this kernel version correspond to Windows 8 or Server 2012, not 10. EDIT Ok, this is a known behavior. See here and here. So this has nothing to do with the error.
Trying to reproduce the bug without success on my side. I suspect more and more a bad configuration on the host machine or network.
As seen on the code above, the RestClient object is instantiated only once and used everywhere in the application. Could it be an issue?
Investigation continues.
Related
I'm trying to access my RESTful API to retrieve data from a MySQL database. Everything is set up and works perfectly on my C# WPF project. But when using the exact same code in Xamarin Forms (built for Android) I cannot get a successful SSL handshake with my server.
Server details
Let's Encrypt SSL certificate (definitely valid)
Apache BasicAuth (.htaccess)
HTTPS only (Rewrite HTTP on), so port 443
REST API: php-crud-api (by mevdschee) to access MariaDB 10.3
I'm using Flurl.Http (uses HttpClient) to establish the connection, but get an exception on jsonReader.Wait():
var jsonReader = "https://example.com/my_api/api.php/records/my_table?order=id,desc&size=10"
.WithHeader("Accept", "application/json")
.WithBasicAuth("username", "password")
.GetJsonAsync<JsonRootObject>();
// Wait for completion.
jsonReader.Wait();
This is my AggregateException:
System.AggregateException: One or more errors occurred. (Call failed. An error occurred while sending the request
GET https://example.com/my_api/api.php/records/my_table?order=id,desc&size=10) ---> Flurl.Http.FlurlHttpException: Call failed. An error occurred while sending the request
GET https://example.com/my_api/api.php/records/my_table?order=id,desc&size=10 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request
---> System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException:
Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /Users/builder/jenkins/workspace/xamarin-android-d16-1/xamarin-android/external/mono/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00038] in <74989b9daab94528ac2c4b7da235b9c5>:0
at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000a1] in <74989b9daab94528ac2c4b7da235b9c5>:0
at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <74989b9daab94528ac2c4b7da235b9c5>:0
at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000ff] in <74989b9daab94528ac2c4b7da235b9c5>:0
at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x0008b] in <74989b9daab94528ac2c4b7da235b9c5>:0
What I've tried / what I know
Code works perfectly in WPF
Exact same code (copy-paste) does not work in Xamarin Forms (tested on Android Pie)
does not work in emulator nor dedicated device
accessing the REST API via browser works and delivers results as expected (tested on my PC, the emulator in Chrome and my dedicated Android)
changing the HttpClient implementation or SSL/TLS implementation under Project properties -> Android Options -> Advanced does not help
Why is my SSL handshake failing? What is Xamarin doing differently than WPF?
It turned out to be an Apache configuration error. I checked my domain and noticed an error. This was fixed by adding the certificate's chain-file in my Apache Directives:
SSLCertificateFile /etc/letsencrypt/live/example.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem
The last line was the missing one. Weird that browsers and WPF trusted the certificate even without the chain-file. Anyway, now it also works on Android.
In the meantime I've been using this piece of code to disable the SSL validation in my app:
// This disables the SSL certificate validity check.
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
That code is unsafe and should only be used for testing.
your Flurl.Http can fail due to the usage of 2 different project types (.net standard vs class library - I had exactly the same problem with http client I wrote)
try a different http client for .net standard.
in any case, you can write your own http client and debug - that is my recommendation.
if you still stuck - write back I'll help.
Let me try to describe the situation..
Site A:
Is meant is an catch all site, with only 1 library. This is used by an third party software, that send all the items to 1 location with specified information. Behind this is an timerjob, that sends all the items to their respective sites/libraries. (so doesn't stay on Site A)
Site X:
When an new item is added through the timerjob on Site A. An workflow is started, however the workflow is randomly sunspended with a the same error. A manual start on the same item will allways succeed. Thus only the autostart that gets suspended.
Workflow is made in visual studio and is widely used in the application without problems.
The error message:
Details: An unhandled exception occurred during the execution of the
workflow instance. Exception details: System.ApplicationException:
HTTP 401 {"error_description":"The server was unable to process the
request due to an internal error. For more information about the
error, either turn on IncludeExceptionDetailInFaults (either from
ServiceBehaviorAttribute or from the configuration
behavior) on the server in order to send the exception information
back to the client, or turn on tracing as per the Microsoft .NET
Framework SDK documentation and inspect the server trace logs."}
{"x-ms-diagnostics":["3001000;reason=\"There has been an error
authenticating the
request.\";category=\"invalid_client\""],"SPRequestGuid":["c0fb74e8-6b48-0deb-85c4-7fdab324d2f7"],"request-id":["c0fb74e8-6b48-0deb-85c4-7fdab324d2f7"],"X-FRAME-OPTIONS":["SAMEORIGIN"],"SPRequestDuration":["17"],"SPIisLatency":["0"],"Server":["Microsoft-IIS/8.5"],"WWW-Authenticate":["Bearer
realm=\"97b27fa4-494a-4bff-a6a0-00b798335a86\",client_id=\"00000003-0000-0ff1-ce00-000000000000\",trusted_issuers=\"00000005-0000-0000-c000-000000000000#*,00000003-0000-0ff1-ce00-000000000000#97b27fa4-494a-4bff-a6a0-00b798335a86\"","Negotiate","NTLM"],"X-Powered-By":["ASP.NET"],"MicrosoftSharePointTeamServices":["15.0.0.4687"],"X-Content-Type-Options":["nosniff"],"X-MS-InvokeApp":["1;
RequireReadOnly"],"Date":["Mon, 14 Jan 2019 13:47:12 GMT"]} at
Microsoft.Activities.Hosting.Runtime.Subroutine.SubroutineChild.Execute(CodeActivityContext
context) at
System.Activities.CodeActivity.InternalExecute(ActivityInstance
instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at
System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor
executor, BookmarkManager bookmarkManager, Location resultLocation)
Tried many given solutions (User Sync (however manual start with the same user is not a problem)), Workflow App Permissions feature ..
We are connecting to a specific website using a TLS connection. We are enforcing TLS 1.2 as the only allowed protocol in the client code. In about 99% of all cases connection setup is OK and shows no errors. However from time to time we receive the following exception:
System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: The buffers supplied to a function was too small
--- End of inner exception stack trace ---
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at Fw.Connections.Ssl.Private.ClientInstance.AuthenticateClientCallback(IAsyncResult result);
We have tested the remote server with SSL labs and we receive an A+ with no failures or problems reported, it's not a server under our control so the remote environment is not very well known.. We have tested the client code on Windows server 2012 (datacenter) and Windows7 clients. Both show the error from time to time. Is this a known bug in the .NET framework?
I'm trying to enable ssl for my WCF, I'm doing exact steps from this tutorial:
https://github.com/geersch/WcfEnableSsl
I even tried to run example application (https://github.com/geersch/WcfEnableSsl)
but in my app and in the example app i get
fault exception was unhandled
The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework SDK documentation and inspect the server trace logs.
at this line
var customers = proxy.GetCustomers();
can anyone help with this?
EDIT
I added a printscreen of an error with details
Hi I am new to ServiceStack+Redis. Now I am looking at RedisStackOverflow example that comes with ServiceStack.
I get an error when I try to run it:
SocketException - An operation was attempted on something that is not
a socket.
Call stack location: ServiceStack.Redis.PooledRedisclientManager.Finalize() Line 324
The web page shows up but with no data. I also have Redis as Windows Services 2.4.6 32bit installed and verified it is working with the Redis Client.
I am running on a vista 32bit box, iis7 vs2010:
Sorry about the picture size, please zoom in to see the exception details.