We Keep Coding

Azure APIM URL throws System.Net.WebException - How SSL/TLS making difference in azure web api and azure APIM?

I have created web api and tried to issue GET request using c# as follow
namespace APIMCheck
{
class Program
{
static void Main(string[] args)
{
string thumbprint = "***";
string url #"https://******-us-stats-webapi.azurewebsites.net/statistics/v1/masterData/carTypes";
X509Store store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadOnly);
X509Certificate2Collection certificates = store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, true);
X509Certificate2 certificate = certificates[0];
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.ClientCertificates.Add(certificate);
req.Method = WebRequestMethods.Http.Get;
Console.WriteLine(Program.CallAPI(req).ToString());
Console.Read();
}
public static string CallAPI(HttpWebRequest req)
{
var httpResponse = (HttpWebResponse)req.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
return streamReader.ReadToEnd();
}
}
public static bool AcceptAllCertifications(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certification, System.Security.Cryptography.X509Certificates.X509Chain chain, System.Net.Security.SslPolicyErrors sslPolicyErrors)
{
return true;
}
}
}
I get response with data. All good.
Now, I have created Azure APIM, which will act as front end for above web API
This is policy configured in Azure API Management portal
<policies>
<inbound>
<base />
<choose>
<when condition="#(context.Request.Certificate.Verify() != true || context.Request.Certificate == null || context.Request.Certificate.Issuer != "CN=MySubCA, DC=MYEXT, DC=NET" || context.Request.Certificate.NotAfter < DateTime.Now)">
<return-response>
<set-status code="403" reason="Invalid client certificate" />
<set-body template="none">#(context.Request.Certificate.Issuer.ToString())</set-body>
</return-response>
</when>
</choose>
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
Now, changed the url as follow to point apim
string url = #"https://******-us-stats-apim.azure-api.net/statistics/v1/masterData/carTypes";
I get below error
The request was aborted: Could not create SSL/TLS secure channel for HttpWebRequest
How SSL/TLS making difference in web api and APIM?
Anything to do with firewall?

By default, TLS 1.2 is enabled for Azure API Management gateway.
You could go to your azure api management(on portal) > Protocol settings> turn on tls 1.2 and ssl 3.0.

There are a few things you do in your test app and APIM cannot do on it's own:
Enable SSL3
Add client certificate
Ignore any SSL validation errors.
Of the things above only #2 is fine to have in production. SSL3 is deprecated and should not be used in production. Ignoring any SSL errors is also unwise, since that way you cannot be sure that you're talking to your server.
Now assuming that you're fine with all of the above:
For SSL3 follow #Joey Cai answer.
To allow APIM to use your certificate to authenticate to backend follow this guidance: https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-mutual-certificates
To control SSL validation at APIM level use this REST API: https://learn.microsoft.com/en-us/rest/api/apimanagement/2019-01-01/backend/createorupdate and set-backend-service policy.
The main reason for #2 and #3 is that client-to-APIM and APIM-to-backend are two separate connections. So when APIM needs to make a call to backend it must have client certificate available (if it's required by backend). That also means that by default APIM will not require client to provide certificate.

"The request was aborted: Could not create SSL/TLS secure channel" in HttpWebRequest, even after specifying ServicePointManager.SecurityProtocol

In order to meet PCI compliance, we have had to completely disable TLS 1.0 from our servers. Ever since doing this, I have had to specify which protocol to use when making web requests in order for things to work. However, one particular problem persists. Even when I specify the protocol, I still get the error stated in the title.
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
httpWebRequest.KeepAlive = false;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11;
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
streamWriter.Write(body);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
return result;
}
The error itself is generated from the line -
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
As you can see, I am specifying to use either TLS 1.1 or 1.2 (I originally had it at 1.2 only, but thought allowing 1.1 as well would help, but it did not).
Additionally (not sure if this is pertinent or not), this particular snippet of code is always calling an API that we also run and maintain, so essentially, the server is calling itself to get a piece of info it needs.
And I am fast running out of ideas on what I can try to further debug or fix this, so any help out there would be greatly appreciated.

I finally figured this one out, and the solution is far simpler than I'd like to admit. I simply moved the line
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11;
to the very top of the code snippet I posted in the question, and everything has been working since. I'm guessing it needs to be in place before creating the HttpWebRequest?

.Net core HttpClient bug? SocketException: An existing connection was forcibly closed by the remote host

The following code runs without any error in a full .Net framework console program. However, it got the following error when running in .Net core 2.1.
System.AggregateException
HResult=0x80131500
Message=One or more errors occurred.
Source=System.Private.CoreLib
StackTrace:
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at ConsoleApp1.Program.requestString(String url) in C:\source\repos\ConsoleApp1\Program.cs:line 38
at ConsoleApp1.Program.Main(String[] args) in C:\source\repos\ConsoleApp1\Program.cs:line 13
Inner Exception 1:
HttpRequestException: The SSL connection could not be established, see inner exception.
Inner Exception 2:
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Inner Exception 3:
SocketException: An existing connection was forcibly closed by the remote host
Code:
class Program
{
static void Main(string[] args)
{
var url = "https://google.com";
var (statusCode, html) = requestString(url);
Console.WriteLine("%d %s", statusCode, html);
}
static CookieContainer cc = new CookieContainer();
static HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = false, CookieContainer = cc };
public static async Task<(int statusCode, string content)> requestStringAsync(string url)
{
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
// | SecurityProtocolType.Ssl3;
using (var request = new HttpRequestMessage { RequestUri = new Uri(url), Method = HttpMethod.Get })
using (var client = new HttpClient(handler))
{
var response = await client.SendAsync(request); // Error (actual line)
// response.EnsureSuccessStatusCode() |> ignore
var statusCode = (int)response.StatusCode;
var content = await response.Content.ReadAsStringAsync();
return (statusCode, content);
}
}
public static (int statusCode, string content) requestString(string url)
{
return requestStringAsync(url).Result;
}
}

There is a bug for .NET Core 2.1 Preview mentioning this issue. That may be the cause. However, I also notice that your setting of TLS is incorrect. You're currently enabling it, but overwriting all other protocols that have been set. Instead of this:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
You should be using this:
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
// ----------------------------------^
I think that's a side issue, but worth fixing all the same.
Update
The referenced GitHub issue above has a discussion which eventually links to the official announcement for .NET Core 2.1 SDK Preview 2. It has the following to say:
Sockets Performance and SocketsHttpHandler
We made major improvements to sockets in .NET Core 2.1. Sockets are the basis of both outgoing and incoming networking communication. The higher-level networking APIs in .NET Core 2.1, including HttpClient and Kestrel, are now based on .NET sockets. In earlier versions, these higher-level APIs were based on native networking implementations.
...
You can use one of the following mechanisms to configure a process to use the older HttpClientHandler:
From code, use the AppContext class:
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
The AppContext switch can also be set by config file.
The same can be achieved via the environment variable DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER. To opt out, set the value to either false or 0.

In my case the root cause was using multiple concurrent calls of HttpClient.GetAsync(url). I fixed it by making HttpClient instance as singleton.
More info here.
Therefore, HttpClient is intended to be instantiated once and reused throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. That issue will result in SocketException errors. Possible approaches to solve that problem are based on the creation of the HttpClient object as singleton or static...

https request fails only in .net web app

I am trying to patch a .net web application that after years of working started failing to get UPS shipping quotes, which is impacting web business dramatically. After much trial and error, I found the following code that works just fine in a console application:
static string FindUPSPlease()
{
string post_data = "<xml data string>";
string uri = "https://onlinetools.ups.com/ups.app/xml/Rate";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
byte[] postBytes = Encoding.ASCII.GetBytes(post_data);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postBytes.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
// get response and send to console
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
Console.WriteLine(response.StatusCode);
return "done";
}
This runs in Visual Studio just fine and gets a nice little response from UPS that the XML is, of course, malformed.
But, if I paste this function into the web application without changing a single character, an exception is thrown on request.GetRequestStream():
Authentication failed because the remote party has closed the transport stream.
I tried it in a couple of different place in the application with the same result.
What is there about the web application environment that would affect the request?

It turns out to be a TLS issue. I guess the console app uses a higher protocol by default than the web application, although none was specified. So, all you have to do is add the following line(s) of code sometime prior to making the request:
using System.Net;
...
System.Net.ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
That was all it took, though I spent an enormous amount of getting there.
Here is the response from UPS on the issue:
Effective January 18, 2018, UPS will only accept TLS 1.1 and TLS 1.2 security protocols... 100% of requests from customers who are on TLS 1.0 while using production URLS (onlinetools.ups.com/tool name) will be rejected.
Anyway, hope this helps someone.
Jim

Can you try setting the Credentials to your request object like following.
request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;

Try setting the default credentials or check if there is any proxy server set and pass it like in the example below.
The example is given for WebClient.
I was having problem with setting Default Credential, as proxy was enabled on the server. So i passed the proxy URL and port with credentials which can access it.
using (System.Net.WebClient web = new System.Net.WebClient())
{
//IWebProxy defaultWebProxy = WebRequest.DefaultWebProxy;
//defaultWebProxy.Credentials = CredentialCache.DefaultCredentials;
//web.Proxy = defaultWebProxy;
var proxyURI = new Uri(string.Format("{0}:{1}", proxyURL, proxyPort));
//Set credentials
System.Net.ICredentials credentials = new System.Net.NetworkCredential(proxyUserId, proxyPassword);
//Set proxy
web.Proxy = new System.Net.WebProxy(proxyURI, true, null, credentials);
web.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
var result = web.UploadString(URL, "");
return result;
}

WebClient is very slow

I have problem with Webclient.
It is very slow. It takes about 3-5 seconds to downloadString from one website.
I don't have any network problems.
This is my Modifed WebClient.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
namespace StatusChecker
{
class WebClientEx: WebClient
{
public CookieContainer CookieContainer { get; private set; }
public WebClientEx()
{
CookieContainer = new CookieContainer();
ServicePointManager.Expect100Continue = false;
Encoding = System.Text.Encoding.UTF8;
WebRequest.DefaultWebProxy = null;
Proxy = null;
}
public void ClearCookies()
{
CookieContainer = new CookieContainer();
}
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = CookieContainer;
}
return request;
}
}
}
UPDATE:
In wireshark I saw that single DownladString is sending and receiving few thousands packets.

There may be two issues at hand here (that I've also noticed in my own programs previously):
The first request takes an abnormally long time: This occurs because WebRequest by default detects and loads proxy settings the first time it starts, which can take quite a while. To stop this, simply set the proxy property (WebRequest.Proxy) to null and it'll bypass the check (provided you can directly access the internet)
You can't download more than 2 items at once: By default, you can only have 2 simultaneous HTTP connections open. To change this, set ServicePointManager.DefaultConnectionLimit to something larger. I usually set this to int.MaxValue (just make sure you don't spam the host with 1,000,000 connections).

There are a few options if it is related to the initial proxy settings being checked:
Disable the automatic proxy detection settings in Internet Explorer
Set the proxy to null:
WebClient.Proxy = null
On application startup set the default webproxy to null:
WebRequest.DefaultWebProxy = null;
In older .NET code instead of setting to null, you used to write this (but null is now preferred):
webclient.Proxy = GlobalProxySelection.GetEmptyWebProxy();

Maybe it will help somebody. Some web services support compression (gzip or other). So you can add Accept-Encoding header for your requests and then enable automatic decompression for web client instance. Chrome works in that way.