We Keep Coding

HTTP Web Request TLS 1.3 with C# .NET Core 3.1 exception "The client and server cannot communicate, because they do not possess a common algorithm."

When attempting a web request to a site which requires TLS 1.3 https://shop.claytonengineering.com/. I'm receiving the following exception on "request.GetResponse();".
Exception: The SSL connection could not be established, see inner exception.
Inner Exception: The client and server cannot communicate, because they do not possess a common algorithm.
From Google Chrome Developer tools Security tab - "The connection to this site is encrypted and authenticated using TLS 1.3, X25519, and AES_128_GCM."
Any ideas on how to get this request to work?
HttpWebResponse response = null;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://shop.claytonengineering.com/");
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36";
request.Headers.Add("Sec-Fetch-Mode", #"navigate");
request.Headers.Add("Sec-Fetch-User", #"?1");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
request.Headers.Add("Sec-Fetch-Site", #"same-origin");
request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls13;
response = (HttpWebResponse)request.GetResponse();
string html = null;
using (StreamReader stream = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
html = stream.ReadToEnd();
}

There are at least two parts to a successful TLS connection:
Protocol version
Ciphersuite match
In both cases both the client and server must match up to use the same protocol and cipher. You already understand that there are multiple TLS versions (1.0, 1.1, 1.2, 1.3) but there are also multiple ciphers that are used to encrypt/decrypt information.
After the 3-way handshake, the client sends its accepted versions in the "Client Hello" and a list of accepted ciphers to the server. The server finds a match and sends that information back to the client ("Server Hello"), then communication begins.
In this case, although you're both using TLS 1.3, you aren't matching up with a cipher. The server has a list of ciphers that can be found by using the test found at ssllabs.com.
The list of ciphers accepted by the server you listed above is in the image below. You can enable one of the above on your client and then the connection should succeed. I assume you're using Windows, so please see this:
https://www.howtogeek.com/221080/how-to-update-your-windows-server-cipher-suite-for-better-security/#:~:text=On%20the%20left%20hand%20side%2C%20expand%20Computer%20Configuration%2C,%E2%80%9CEnabled%E2%80%9D%20button%20to%20edit%20your%20server%E2%80%99s%20Cipher%20Suites.

I had this very same issue and found out that while .Net framework 4.8 and .Net5.0 and 6.0 all have implementations of the TLS1.3 stack, none of them seem to use them when you do not tell Windows to do so.
In my case I had to add the following registry key:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
For some reason these are not there by default.

Calling a Restful Web Service that has HttpRequestMessage as a parameter?

I am in the process of migrating one of my company's web services to a new server, and unfortunately the previous developers left us no way to test the migration of this service prior to migrating the production version. This leaves us in a harsh situation where I have to formulate a backup plan in case things go wrong when we migrate to the new server.
To understand my plan, you must first understand that the flow of execution for this web service is currently:
Customer calls platform.
Platform calls web service.
Web service responds to platform.
Platform responds to customer.
Simple enough, but the platform's changes are already in place for deployment at the flip of a switch and the developer will not be in house for the migration. Thus, they will flip the switch and leave me hoping the migration works.
I have a simple rollback plan in which the platform's developer won't be required for me to rollback. I simply inject a middle-man to the chain above which acts as a conduit to the web service for the platform:
Customer calls platform.
Platform calls conduit service.
Conduit service calls web service.
Web service responds to conduit.
Conduit responds to platform.
Platform responds to customer.
This way, if for some reason, the migrated version of the web service fails, I can fallback to the original version hosted on the old server until we can investigate what's missing and why it all went wrong (currently we have no way to do this).
Now that you have an understanding of the issue, I have a simple issue with writing the conduit to the underlying web service. I encountered a method in the web service that returns HttpResponseMessage and expects HttpRequestMessage as a request. This is rather confusing since the platform calls this method via the following URI:
test.domain.com:port/api/route/methodname
I have no access to the code under this URI assignment (which is in RPG code), so I have no idea how they are passing the data over. Currently my code is simple:
[Route("MethodName")]
[HttpPost]
public HttpResponseMessage MethodName(HttpRequestMessage request) {
try {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create($"{ServiceRoute}/api/route/methodname");
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return response; // There is a type mismatch, I know.
} catch (Exception e) {
// Log exception.
return null;
}
}
How can I call a restful web service and pass on the request message to the service?
NOTE: I understand, the snippet I've supplied will not work and has an error. I DO NOT expect anyone to just hand out code. References and explanations as to what needs to be done and why are what I'm looking for.

I'm not sure I totally understand the question, so apologies if this isn't helpful, but if your conduit truly just forwards each request as-is, you should be able to reuse the incoming HttpRequestMessage by changing the RequestUri property to the web service URI and forwarding it to the web service with an instance of HttpClient. Something like this:
[Route("MethodName")]
[HttpPost]
public async HttpResponseMessage MethodName(HttpRequestMessage request) {
request.RequestUri = $"{ServiceRoute}/api/route/methodname";
request.Method = HttpMethod.Get;
request.Headers.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
//add any additional headers, etc...
try
{
//best practice is to reuse the same HttpClient instance instead of reinstantiating per request, but this will do for an example
var httpClient = new HttpClient();
var response = await httpClient.SendAsync(request);
//perform any validation or modification of the response here, or fall back to the old web service on a failure
return response;
}
catch (Exception e)
{
// Log exception.
return null;
}
}

Https GET request passes in Postman and browser but fails in code

I'm trying to obtain a JSON via a rest API using, targeting .Net 4.5
I've tried various methods in code, but the all end up in me getting:
"Authentication failed because the remote party has closed the
transportstream" .
the exact same URL works via browser and Postman.
So far, I've tried using .Net's WebClient, HttpClient and HttpWebRequest with identical results. I've tried comparing requests between Postman and my code (via RequestBin), but even when they were identical, I still kept getting back:
Authentication failed because the remote party has closed the
transport
My current code is using HttpWebRequest, but every solution will do.
I've played around with all of the security protocols, some of them will cause the API to return 404 and some will cause the server to return
"Authentication failed because the remote party has closed the
transport stream".
Here's my current code:
public string GetCityStreets()
{
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var request = (HttpWebRequest)WebRequest.Create(url);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls12 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls;
var response = (HttpWebResponse)request.GetResponse();
string jsonResponse;
using (var reader = new StreamReader(response.GetResponseStream()))
{
jsonResponse = reader.ReadToEnd();
}
return jsonResponse;
}
In my current code, the exception is thrown when the request is actually made: request.GetResponse().
What I need, essentially, is to get the JSON from the API.

Set SecurityProtocolType.Tls12 before you initalize the request:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12`
var request = WebRequest.CreateHttp(url);
If you're on Windows 7. On Windows 10, you should only need SecurityProtocolType.SystemDefault.
Note: To enable TLS1.3 (it's available in both Windows 7 and Windows 10), If you don't use .Net 4.8 or .Net Core 3.0, since there's no enumerator for it, you can set it with:
var secProtoTls13 = (SecurityProtocolType)12288;
Remove all the other SecurityProtocolType you have set there.
Setting the User-Agent header is also mandatory, otherwise you will receive a 404 (not found). Here's the FireFox header:
request.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0";
A note on the User-Agent header: this specific site doesn't activate HTTP Strict Transport Security (HSTS). But some sites do, when they see that the WebBrowser supports it. HttpWebRequest doesn't understand it, so it will simply wait for a response that never comes, since the Site is waiting for interaction.
You may want to use the IE11 header instead.
Also add this other header:
request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");

The server side appears to be checking the user agent (presumably to stop bots and other code (like yours!) from hitting the endpoint). To bypass this, you will need to set the user agent to a value such that it thinks you are a web browser.
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";
works, as an example.
You may wish to consider setting the ServicePointManager.SecurityProtocol just once (at app startup) rather than on each request.

This works using HttpClient (.net 4.5 and up)
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", "C# App");
Task<string> response = client.GetStringAsync(url);
Console.WriteLine(response.Result);
Think the server requires a user agent.

Windows Store Application Http Requests Only Work After Manifest Change

I am currently building a windows store application and I am running into what I now think is a bug.
All my http requests fail except when I make a change to the application manifest then they will work on the first run but straight after that the next web request will fail.
The strange thing is that in order for me to get it to work again I will have to remove a capability from the manifest, even if it is an important one such as the internet capability then the application will work!
Here are the headers I am passing in my HttpClient request:
requestMessage.Headers.TryAddWithoutValidation("Connection", "keep-alive");
requestMessage.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, sdch");
requestMessage.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
requestMessage.Headers.TryAddWithoutValidation("Host", "xx.xx.x.xxx");
requestMessage.Headers.TryAddWithoutValidation("Cache-Control", "no-cache");
requestMessage.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.107 Safari/537.36");
requestMessage.Headers.TryAddWithoutValidation("Upgrade-Insecure-Requests", "1");
Even when I get a web request to hit the server it will always fail on the next call.
Here is the exception I get when it fails:
InnerException = {"An attempt was made to access a socket in a way forbidden by its access permissions 10.98.0.181:80"}
We have deployed the Web API services on IIS and the application is running on a tablet that connects to the server using a VPN.
The tablet is running Symantic Endpoint Protection which is managed externally.
Is there a caching option I can turn of on the device that could be causing this or a setting I have overlooked?

C# - User Agent for Web in WP8.1 Apps

One question that's been confusing me and could really do with some insight.
I need to retreive Json objects from a http service. When I tested this in a Console Window, I kept receiving a "Internal Server Error : 500" until I set the UserAgent property for the WebClient object.
Example:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36");
content = client.DownloadString(url);
Now, if I need to do the same for a WP8.1 app, how would I detect (if I need to in the first place?) the UserAgent (and set it) and be able to retrieve the data?
Thank you all.

Windows Phone 8.1 App will use HttpClient. By default there will not be a user agent set. The default user-agent for the phones web browser is:
"Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 520) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537"
You can manually set the user-agent on the HttpRequestMessage.Headers.UserAgent property.
References:
HttpClient
https://msdn.microsoft.com/en-us/library/windows/apps/xaml/windows.web.http.headers.httprequestheadercollection.aspx
User-Agent
https://msdn.microsoft.com/en-us/library/ie/hh869301(v=vs.85).aspx#ie11\

The class libraries for using http do not add any User Agents by default. See these lines from the msdn page:
By default, no user-agent header is sent with the HTTP request to the web service by the HttpClient object. Some HTTP servers, including some Microsoft web servers, require that a user-agent header be included with the HTTP request sent from the client. The user-agent header is used by the HTTP server to determine how to format some HTTP pages so they render better on the client for different web browsers and form factors (mobile phones, for example). Some HTTP servers return an error if no user-agent header is present on the client request. We need to add a user-agent header to avoid these errors using classes in the Windows.Web.Http.Headers namespace. We add this header to the HttpClient.DefaultRequestHeaders property.
For more details, refer the link below:
How to connect to an HTTP server using Windows.Web.Http.HttpClient (XAML)
Also look at the answer below (by Bret Bentzinger) for the exact user agent string.