I'm struggling to connect to a REST web service that's working only over HTTPS / SSL from my .NET application.
I received the certificate and private key to use as two separate files - a certificate.pem file which contains the certificate, and the webservice.key file which contains the private key. Those are both text files with BASE64 encoded binary data contained in them.
The provider also sent me a PDF showing how to call that web service using CURL and those two files, and that works just fine:
curl.exe -k -v "https://(URL)" --cert certificate.pem --key webservice.key
I need to use the -k option since there seems to be a self-signed certificate somewhere in the hierarchy of certs. Without this option, the call fails.
In order to call this web service from a .NET application (a console app for now), I used OpenSSL (on Windows) to combine these two files into a *.pfx file using this command:
openssl pkcs12 -export -out webservice.pfx -in certificate.pem -inkey webservice.key
This seems to have worked, too - no errors were reported, the file was created and is about 3K in size and it's a totally binary file.
Now, I tried to call that web service from my .NET code something like this:
try
{
// use the SSL protocol (instead of TLS)
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
// ignore any certificate complaints
ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => { return true; };
// create HTTP web request with proper content type
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.ContentType = "application/xml;charset=UTF8";
// grab the PFX as a X.509 certificate from disk
string certFileName = Path.Combine(certPath, "webservice.pfx");
// load the X.509 certificate and add to the web request
X509Certificate cert = new X509Certificate(certFileName, "(top-secret password)");
request.ClientCertificates.Add(cert);
request.PreAuthenticate = true;
// call the web service and get response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
}
catch (Exception exc)
{
// log and print out error
}
However, I can try whatever I like (fiddling around with various settings, on the ServicePointManager and the HttpWebRequest, but I just keep getting these errors:
WebException: The underlying connection was closed: An unexpected error occurred on a send.
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
and no response - even though communicating with the service with CURL has worked just fine.....
What am I missing?? I'm a bit puzzled and mystified by all those certificates, private keys, service point manager options and so on - just waaaaay too many knob and switches to turn, set or turn off - what are the RIGHT settings here??
Update:
If I use
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
then the error just simply is:
WebException: The request was aborted: Could not create SSL/TLS secure channel.
S O L U T I O N :
In the end, with looking at the output from curl and a lot of help from #Alexandru and #JurajMajer, I was able to get this to work with this code:
try
{
// use the TLS protocol
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
// create HTTP web request with proper content type
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.ContentType = "application/xml;charset=UTF8";
// grab the PFX as a X.509 certificate from disk
string certFileName = Path.Combine(certPath, "webservice.pfx");
// load the X.509 certificate and add to the web request
X509Certificate2 cert = new X509Certificate2(certFileName, "(top-secret password)");
request.ClientCertificates.Add(cert);
request.PreAuthenticate = true;
// call the web service and get response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
string xmlContents = new StreamReader(responseStream).ReadToEnd();
}
catch (Exception exc)
{
// log and print out error
}
You've used the X509Certificate(String, String) constructor with a PKCS#12 certificate, but that constructor only works for PKCS#7 certificates, as MSDN says it...
Initializes a new instance of the X509Certificate class using the name
of a PKCS7 signed file and a password to access the certificate.
PKCS#7 does not include the private (key) part of a certificate/private-key pair, which you will need. This means you will need to use your PKCS#12 certificate given the nature of your certificate.
You may want to try the X509Certificate2(String, String) constructor with your existing PKCS#12 certificate, as this constructor is used with PKCS#12 (PFX) files that contain the certificate's private key, as MSDN says...
This constructor creates a new X509Certificate2 object using a
certificate file name and a password needed to access the certificate.
It is used with PKCS12 (PFX) files that contain the certificate's
private key. Calling this constructor with the correct password
decrypts the private key and saves it to a key container.
Try to enable Network Tracing in App.config on the client - instructions here. That should create network.log with more debug info. In my test environment I have one pfx which works and one which doesn't.
network.log for working pfx:
SecureChannel#9343812 - We have user-provided certificates. The server has specified 34 issuer(s). Looking for certificates that match any of the issuers.
SecureChannel#9343812 - Left with 1 client certificates to choose from.
SecureChannel#9343812 - Trying to find a matching certificate in the certificate store.
SecureChannel#9343812 - Locating the private key for the certificate:
SecureChannel#9343812 - Certificate is of type X509Certificate2 and contains the private key.
network log for non-working pfx:
SecureChannel#26756241 - We have user-provided certificates. The server has specified 34 issuer(s). Looking for certificates that match any of the issuers.
SecureChannel#26756241 - Left with 0 client certificates to choose from.
So for me the problem is my non-working certificate was issued by CA not in list.
Interesting points (possible problems):
1.) Server sends the list of known issuers for client certificate.
2.) Client code is looking for certificate and private key in certificate store event though both are in pfx.
Related
I have created a C# Web Server and I have some problems with the SSL/TLS certificates.
For the setup I have changed my /etc/hosts file for that any request to example.com will direct
to localhost. After that I have created a TCP server which works quite as expected. For the
HTTPS I added a System.Net.Security.SslStream layer with a default self-signed certificate which
also works as expected.
But for now I want to send a certificate that depends on the location the user requests. My server
code is like this:
var stream = new SslStream(
innerStream: client.GetStream(),
leaveInnerStreamOpen: false,
userCertificateValidationCallback: null,
userCertificateSelectionCallback: LocalCertificateSelection
);
stream.AuthenticateAsServer(
// I am forced to provide a default certificate. null will throw an error
serverCertificate: Settings.Certificate,
clientCertificateRequired: false,
enabledSslProtocols: SslProtocols.None,
checkCertificateRevocation: true
);
private X509Certificate LocalCertificateSelection(
object sender,
string targetHost,
X509CertificateCollection localCertificates,
X509Certificate remoteCertificate,
string[] acceptableIssuers
)
{
// at this point: targetHost == null && acceptableIssuers.Length == 0
return Settings.Certificate;
};
But if I open the page in Firefox or in curl the targetHost and acceptableIssuers from my
certificate selection callback are always empty.
How am I supposed to get the host name to provide the correct certificate?
Edit 1:
My library is targeted for .Net Standard 2.1. Maybe this will help.
Edit 2:
I have added the certificate to the key store of the server (as it was mentioned in a comment), but this wont resolve the problem:
using var store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadWrite);
store.Add(Settings.Certificate);
After that I have captured some packages with Wireshark and I have noticed the following:
After TCP connection is finished:
Client sends a TLS Hello request with a valid SNI extension (here is the host name I want to get)
Debugger break point in my code is called. The SslStream tries to get a valid certificate: targetHost and acceptableIssuers are empty.
Server sends the default certificate with possible invalid host name
I'm trying to communicate with an external website using mutual certificate authentication, but am receiving a "Could not establish secure channel for SSL/TLS with authority" exception. I've narrowed it down to the following snippet (sensitive things removed):
void Example()
{
string KeyIdentifier = "<MyKeyId>";
X509Store store = new X509Store("My", StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadOnly);
X509Certificate2Collection certificates = store.Certificates.Find(X509FindType.FindByThumbprint, KeyIdentifier, true);
X509Certificate2 certificate = certificates[0];
HttpWebRequest r = (HttpWebRequest)HttpWebRequest.Create("https://www.example.org");
r.ClientCertificates.Add( certificate);
r.Accept = "text/xml";
r.Method = WebRequestMethods.Http.Post;
string result = null;
using (HttpWebResponse resp = (HttpWebResponse)r.GetResponse())
{
StreamReader reader = new StreamReader(resp.GetResponseStream());
result = reader.ReadToEnd();
}
}
I've set up the certificates and such properly and this code finds them successfully, with certificate.HasPrivateKey=true, but fails. SoapUI happily connects with TLS. Chrome reports no certificate errors when I access the website. However, the above code always throws the exception. I've tried all 3 TLS versions with no success. Using ServicePointManager to skip server Cert validation makes no difference.
Strangely, if I specify SSL3
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
then the process completes successfully, but naturally I don't want to be using SSL3. The app is .Net 4.6 running on Server 2012.
After several months and more digging on and off I ended up trying to make the connection using SSLStream, getting a different exception, and finding this answer: A call to SSPI failed, see inner exception - The Local Security Authority cannot be contacted
which advises you to add the following registry key:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman] "ClientMinKeyBitLength"=dword:00000200
This switches the minimum key length for Diffie-Hellman to 512-bit. This minimum value was raised in a Windows Update to 1024-bit to attempt to mitigate logjam attacks, so take care in setting this value as it is system wide and may open up attack vectors against your system.
I'm having real issues attempting to contact a secure rest URL. Basically I have little knowledge of certificates, and am wondering whether what I am doing with the certificate file(s) is correct.
I have 2 URL's effectively one for read and one for write.
I have 2 .pem files, one for each, that were provided with the URLs. The act of importing these into Certificate manager didn't change the outcome below).
I have also a .p12 file, which if I'm honest don't know where it fits in to all this... (I cannot import this into my Win7 cert magager as I do not have a password)
When running req.GetResponse() an exception occurs:
"The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel."
Inner Ex: "The remote certificate is invalid according to the validation procedure."
I'm currently unsure as to whether things need a little tinkering or I'm simply way off.
code is basically...
if (!File.Exists(certificateLocation))
{
throw new Exception(string.Format("The specified certificate file does not exist: {0}", certificateLocation));
}
//Cert Challenge URL
Uri requestURI = new Uri(url);
//Create the Request Object
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(requestURI);
//Set the Request Object parameters
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
req.AllowAutoRedirect = false;
//Create certificate from our file
X509Certificate cert = X509Certificate.CreateFromCertFile(certificateLocation);
req.ClientCertificates.Add(cert);
WebResponse response = req.GetResponse(); // *** Errors here
...
Edit: Currently I'm only attempting the "read" url - seemed logical.
After contacting the issuer for the password accompanying the .p12 certificate, and importing that into the "Trusted Root Certification Authorities section" of certification manager, the error has now changed to the following...
System.Net.WebException
"The underlying connection was closed: An unexpected error occurred on a receive."
Inner Ex:
"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."
-- re-posting comment as answer --
The password is required for this to work.
It is created at the same time as the .p12 file.
Using .NET 4.0 framework I'm trying to make a server to server HTTPS SOAP service call.
On the client side server, I'm trying to add a local SSL certificate so that the server receiving the call could identify me.
The certificate I'm trying to add is not from the local machine's store (the problem persists even if it is), just a (.cer) file placed as a local file on my machine.
The code works fine and seems to add the certificate, but my System.Net.trace.log file keeps adding the following error for every call:
"Cannot find the certificate in either the LocalMachine store or the CurrentUser store." and the server I'm trying to call keeps telling me there's no certificate attached to my request.
The final WebException I get is:
"The request was aborted: Could not create SSL/TLS secure channel."
My GetCertificate function:
private X509Certificate2 GetCertificate()
{
X509Certificate2 cert;
try
{
cert = new X509Certificate2("MyCertificateFoder\\MyCertificate.cer");
}
catch (Exception ex)
{
throw;
}
return cert;
}
Service call code:
X509Certificate2 cert = GetCertificate();
ServiceClient.ClientCertificates.Add(cert);
serviceMethodResponse = ServiceClient.ServiceMethod(serviceMethodRequest);
Will appreciate any advice.
Thanks
The principal mistake here is that when using SSL with client authentication, a .pfx file (with client private key, certificates, certificate chains, root authority certificates) should be used.
NOT the .cer file.
This is the cause of the error.
I have a fully operational system where openssl based clients interact with an openssl server. Each client have its own certificate that is validated by the server. Certificates have been generated with openssl (X509, pem). They are self-signed.
I now want to write a test client based on SslStream. I used the client example from the SslStream class documentation.
My SslStream client is unable to complete the handshake. stunnel complains about the client not sending its certificate. This is confirmed in Wireshark (Certificates Length: 0 in handshake messages).
My client displays the following exception:
Inner exception: The message received was unexpected or badly
formatted
This is how I load my certificate:
X509Certificate cert = new X509Certificate2(filename, password);
X509CertificateCollection certColl = new X509CertificateCollection();
certColl.Add(cert);
I tried retrieving certificate various properties (ex: GetSerialNumberString()). It works. The Verify method returns false. This is the next thing I am going to investigate.
How I setup my SslStream does not seem to matter (same result):
sslStream.AuthenticateAsClient(serverName);
SslStream sslStream = new SslStream(
client.GetStream(),
false,
new RemoteCertificateValidationCallback(ValidateServerCertificate),
new LocalCertificateSelectionCallback(SelectLocalCertificate));
Same with authentication:
sslStream.AuthenticateAsClient(serverName);
sslStream.AuthenticateAsClient(serverName,
certColl,
SslProtocols.Tls,
true);
SelectLocalCertificate gets called (twice) and returns my certificate. ValidateServerCertificate currently never gets called (to my surprise).
How can I debug that? Even better if you can nail my problem.
UPDATE
I have added a function to perform chain validation based on the X509Chain example from the documentation. It displays all sorts of information on the certificate, including two intriguing messages:
Element certificate is valid: False
Element error status length: 1
In the end, I don't really have more details than when I call verify.
The output of openssl verify cert.pem does not report anything unusual.
error 18 at 0 depth lookup:self signed certificate
OK
UPDATE
I extracted the private key and certificate from the pem, and I generated a cert.pfx (pkcs12). I had no issues importing cert.pfx in my personal key store. In my certificate details, I can see a small icon indicating an attached private key.
I modified my client to retrieve the certificate from my personal store. But I am getting the same exact failure.
The solution was to import my CA root certificate on my Windows machine. My client is now able to complete the handshake!
Found the solution by searching for a more complete SslStream example. Thanks to http://geekswithblogs.net/luskan/archive/2007/10/01/115758.aspx.