HttpClient https request fails using hostname, but works using IP address - c#

I am new to web stuff in general. Not sure what the issue is here.
I have a machine with IIS running a ASP NET Core REST API on https.
I can confirm the GET is working via Google Chrome on my machine by doing either
https://test-machine/api/Example (works)
https://10.0.0.21/api/Example (invalid certificate, N.B. this is as expected as the certificate is for "test-machine")
I confirmed the POST action is working using Postman (again ignoring invalid certificate).
Everything is fine till I try and write a client application.
I have the following code.
var ip = Dns.GetHostAddresses("test-machine");
// ip contains the correct IP
using (var hc = new HttpClient())
{
hc.GetAsync(#"https://test-machine/api/Example").Wait();
}
This code fails. I get an System.AggregateException: 'One or more errors occurred.' with 4 exceptions stacked up on top of one another.
HttpRequestException: An error occurred while sending the request.
WebException: The underlying connection was closed: An unexpected error occurred on a send.
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
If I change the code to use the IP instead:
using (var hc = new HttpClient())
{
hc.GetAsync(#"https://10.0.0.21/api/Example").Wait();
}
Then the request works as expected I.e. as with Chrome, I get a certificate problem:
AuthenticationException: The remote certificate is invalid according to the validation procedure.
What am I doing wrong? The Dns is clearly able to get the correct IP, I've seen it in the IPAddress[] returned by GetHostAddresses. Why is the HttpClient not resolving the IP - or is it resolving the IP with another problem?
I have tried
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
and
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
but this makes no difference. Using the
ServicePointManager.ServerCertificateValidationCallback
Also doesn't work as it never reaches the point of calling the callback. It's failing before then.
I suspect the problem is that as I am not familiar with any of this stuff I have missed something important somewhere.
I should note also - this is a .net 4.5.2 Console app. I tried moving to 4.6.1 and this still fails.

I was looking completely in the wrong place. The problem lay in the IIS setup.
The site binding was for some reason explicitly set to 10.0.0.21:433. Switching this to "All Unassigned" - seems to have made this work.
I am probably out of my depth, and could be wrong, but I suspect the reason for this is that according to Wireshark, the response to the MDNS request to find the hostname IP, was returning the ipv6 address of the server machine, which of course isn't 10.0.0.21. Again, I could be wrong.
The reason I am so unsure is that chrome seems to have ploughed right through and found the correct thing when I put in https://test-machine/api/Example and I cannot explain why.

Related

Leaf.xNet.HttpException: "Could not receive the message body of the response HTTP-server 'example.com'"

This error appears only on this link, other links work correctly
Internal exceptions (1/2) - IOException: Unable to read data from the transport connection. An attempt to establish a connection was unsuccessful because the required response was not received from another computer within the required time, or an already established connection was terminated due to an incorrect response from an already connected computer.
Internal exceptions (2/2) - SocketException: An attempt to establish a connection was unsuccessful because the required response was not received from another computer within the required time, or an already established connection was terminated due to an incorrect response from an already connected computer.
I tried to add different headers, cookies, or make a default "clean" request.
Moreover, the postman correctly receives the body, as well as HttpClient.
I will be glad if you help.
For some sites it is not possible to make requests with this framework. Solved the problem by rewriting the code on HttpClient.

C# app can no longer hit website after they switched to TLS 1.2 - failing on GetResponse()

We have an internal application that makes XML calls out to our vendor sites. For PCI compliance and security reasons, they are starting to disable everything except TLS 1.1 and TLS 1.2. They have set up a test site with this new requirement for us to test against.
Our application (C#, Windows Forms app, .NET 4.5) can connect to their current site just fine.
When I try to make a call to their new test site, it fails on the GetRequestStream() call with "The underlying connection was closed: An unexpected error occurred on a send."
If I update the app and set:
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12
Then it gets by the GetRequestStream() and fails on the GetResponse() call with "The underlying connection was closed: The connection was closed unexpectedly."
I have tried a bunch of stuff, none of which has worked. I am setting the UserAgent, I tried setting KeepAlive = false, ...
If I go to the vendors URL in a browser, I can get to it fine. I am just having issues with my C# app
Any thoughts? I obviously dont have access to their server. Is there a setting someone on my machine I need to change?
I think that it is likely that your request is failing during the TLS negotiation stage. I have had similar problems myself and it usually comes down to having to use a network protocol analyzer (such as wireshark) to trace the communication between the client and server and to determine where the messaging has stopped. If TLS negotiation fails, the server will terminate the connection and you will get errors similar to what you got above.
It may be possible that the test server does not have a proper SSL certificate. You can find out in .NET if there are any SSL validation errors using the ServerCertificateValidationCallback. I suggest to ignore the validation errors when in Debug mode, like this:
#if DEBUG
ServicePointManager.ServerCertificateValidationCallback += ValidationCallback;
#endif
var webRequest = HttpWebRequest.Create("https://your.url");
...
#if DEBUG
bool ValidationCallback(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
return true;
}
#endif
This way you knowingly ignore any SSL certificate issues while testing (in Debug mode), but you let the verification mechanism work as usual in production (Release mode). By the way, you can check the sslPolicyErrors parameter to see exactly if and what caused the validation failure.
UPDATE: The author of the question confirmed that the SSL certificate is valid, so my answer doesn't help directly with the error he's receiving.

Strange issue when resolving server SSL certificate within UnitTest in Visual Studio

I have a really strange issue when Visual Studio resolves the server SSL certificate.
The API I am using requires me to send a certificate along with the request to 'verify who I am'. Initially I referred to this certificate within my UnitTest using X509Certificate2 and then importing it as a byte[]. What I didn't want to do, is have this laying around in my code referring to a specific location on my local disk (I want to run the unit tests automatically on my build server) so opted to install the certificate within the certificate store (which I could later install on the build server). And when I tested this Uri within the browser (IE) it asked for the certificate I wanted to use and rather stupidly I clicked OK without really reading what it was asking me to do.
Problem is, since doing this I am unable to run my unit test (I am making an HttpWebRequest) - as soon as it hits the following line it throws a WebException.
using (var response = request.GetResponse())
{
// Removed content ...
}
The exception response is null , here is the exception
System.Net.WebException: The underlying connection was closed: An
unexpected error occurred on a receive. ---> System.IO.IOException:
Unable to read data from the transport connection: An existing
connection was forcibly closed by the remote host. --->
System.Net.Sockets.SocketException: An existing connection was
forcibly closed by the remote host
The only reason I can see for this error is that Visual Studio is using the wrong SSL certificate for the server connection (as I am using the test API Uri, I can verify this is sort of the case by using the live API Uri which works fine)
I have tried the following:
Removing the certificate from the store
Adding the certificate back to the store
Clearing all browser cache
Restoring IE back to 'factory settings'
Clearing the SSL cache within IE
Tried setting Keepalive to true and then false
This issue was affecting all browsers but I have managed to get it working. I have tried to use Fiddler to identify if the server is doing something crazy (including decrypting SSL connections) and I can see this is actually responding with a 403 error.
Has anyone else got any ideas?
Edit
Just tried to run this on the build server, still get the same error so I will check with the API provider to see if a/ my certificate has expired b/ their service is actually up!
System.Net.WebException: The underlying connection was closed: An
unexpected error occurred on a receive. ---> System.IO.IOException:
Unable to read data from the transport connection: An existing
connection was forcibly closed by the remote host. --->
System.Net.Sockets.SocketException: An existing connection was
forcibly closed by the remote host
Issue was with the API provider.

Unable to resolve hostname over vpn

I've been trying get my app to connect to a WCF service within the corporate network.
We use a VPN to keep everything somewhat hidden and secure.
I've tried several different methods and I'm unable to get the phone to connect to the service.
I first added the service as a service reference, and built the client out with the correct URL, but this just did nothing and then failed with a nondescript error message after around 50sec.
I then switched to System.Net.HttpClient. This again failed after around 50sec, but this time it threw a System.Net.WebException with the message "A server with the specified hostname could not be found".
I finally tried the ModernHttpClient as well, and this had the same result as before.
(Note that I tried the first two in a test console app project on my local machine first to make sure that they would complete a request successfully)
To make sure I wasn't going crazy and that the VPN was correctly resolving the name, I created a test page within the app which solely has a WebView on it, with the source set to the service url.
I opened the app and navigated to the test page and it loaded the service definition page without a problem.
for reference, this is the current code I have using ModernHttpClient:
using (var client = new HttpClient(new NativeMessageHandler()))
{
const string soap = "<soap msg>";
client.DefaultRequestHeaders.Add("SOAPAction", "<service namespace>");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("text/xml"));
using (var response = await client.PostAsync("http://<url>/Service.svc", new StringContent(soap, Encoding.UTF8, "text/xml")))
{
using (var content = response.Content)
{
// parse the xml result
}
}
}
Is there something special that I should be doing to tell the HttpClient that it has to use the VPN? I thought that this was handled automatically.
I looked into the VPN configuration to make sure it was correct.
I looked into the DNS configuration to make sure it resolves correctly.
I looked into the phone's configuration to make sure it was correct.
I eventually found the issue is a problem with Airwatch's Tunnel VPN.
The VPN only works the first time the app is launched.
If the VPN remains open and is not closed before the app is restarted, then it will not allow connections to pass through.
Expecting a patch form airwatch soon that should (hopefully) rectify the issue!

Unable to read data from the transport connection : An existing connection was forcibly closed by the remote host

I have a server app and sometimes, when the client tries to connect, I get the following error:
NOTE: the "couldn't get stream from client or login failed" is a text that's added by me in catch statement
and the line at which it stops ( sThread : line 96 ) is :
tcpClient = (TcpClient)client;
clientStream = tcpClient.GetStream();
sr = new StreamReader(clientStream);
sw = new StreamWriter(clientStream);
// line 96:
a = sr.ReadLine();
What may be causing this problem? Note that it doesn't happen all the time
I received this error when calling a web-service. The issue was also related to transport level security. I could call the web-service through a website project, but when reusing the same code in a test project I would get a WebException that contained this message. Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Edit
System.Net.ServicePointManager.SecurityProtocol - This property
selects the version of the Secure Sockets Layer (SSL) or Transport
Layer Security (TLS) protocol to use for new connections that use the
Secure Hypertext Transfer Protocol (HTTPS) scheme only; existing
connections are not changed.
I believe the SecurityProtocol configuration is important during the TLS handshake when selecting the protocol version.
TLS handshake - This protocol is used to exchange all the information required by both sides for the exchange of the actual application data by TLS.
ClientHello - A client sends a ClientHello message specifying the highest TLS protocol version it supports ...
ServerHello - The server responds with a ServerHello message, containing the chosen protocol version ... The chosen protocol version should be the highest that both the client and server support. For example, if the client supports TLS version 1.1 and the server supports version 1.2, version 1.1 should be selected; version 1.2 should not be selected.
This error usually means that the target machine is running, but the service that you're trying to connect to is not available. (Either it stopped, crashed, or is busy with another request.)
In English:
The connection to the machine (remote host/server/PC that the service runs at) was made but since the service was not available on that machine, the machine didn't know what to do with the request.
If the connection to the machine was not available, you'd see a different error. I forget what it is, but it's along the lines of "Service Unreachable" or "Unavailable".
Edit - added
It IS possible that this is being caused by a firewall blocking the port, but given that you say it's intermittent ("sometimes when the client tries to connect"), that's very unlikely. I didn't include that originally because I had ruled it out mentally before replying.
My specific case scenario was that the Azure app service had the minimum TLS version changed to 1.2
I don't know if that's the default from now on, but changing it back to 1.0 made it work.
You can access the setting inside "SSL Settings".
According to "Hans Vonn" replies.
Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
After adding Security protocol and working fine but I have to add before every API call which is not healthy. I just upgrade .net framework version at least 4.6 and working as expected do not require to adding before every API call.
Not sure which of the fixes in these blog posts helped, but one of them sorted this issue for me ...
http://briancaos.wordpress.com/2012/07/06/unable-to-read-data-from-the-transport-connection-the-connection-was-closed/
The trick that helped me was to quit using a WebRequest and use a HttpWebRequest instead. The HttpWebRequest allows me to play with 3 important settings:
and
http://briancaos.wordpress.com/2012/06/15/an-existing-connection-was-forcibly-closed-by-the-remote-host/
STEP 1: Disable KeepAlive
STEP 2: Set ProtocolVersion to Version10
STEP 3: Limiting the number of service points
For those who may find this later, after .NET version 4.6, I was running into this problem as well.
Make sure that you check your web.config file for the following lines:
<compilation debug="true" targetFramework="4.5">
...
<httpRuntime targetFramework="4.5" />
If you are running 4.6.x or a higher version of .NET on the server, make sure you adjust these targetFramework values to match the version of the framework on your server. If your versions read less than 4.6.x, then I would recommend you upgrade .NET and use the newer version unless your code is dependent on an older version (which, in that case, you should consider updating it).
I changed the targetFrameworks to 4.7.2 and the problem disappeared:
<compilation debug="true" targetFramework="4.7.2">
...
<httpRuntime targetFramework="4.7.2" />
The newer frameworks sort this issue out by using the best protocol available and blocking insecure or obsolete ones. If the remote service you are trying to connect to or call is giving this error, it could be that they don't support the old protocols anymore.
Calls to HTTPS services from one of our servers were also throwing the "Unable to read data from the transport connection : An existing connection was forcibly closed" exception. HTTP service, though, worked fine. Used Wireshark to see that it was a TLS handshake Failure. Ended up being that the cipher suite on the server needed to be updated.
This solved my problem. I added this line before the request is made:
System.Net.ServicePointManager.Expect100Continue = false;
It seemed there were a proxy in the way of the server that not supported 100-continue behavior.
This won't help for intermittent issues, but may be useful for other people with a similar problem.
I had cloned a VM and started it up on a different network with a new IP address but not changed the bindings in IIS. Fiddler was showing me "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host" and IE was telling me "Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings". Changing the binding to the new IP address solved it for me.
For some reason, the connection to the server was lost. It could be that the server explicitly closed the connection, or a bug on the server caused it to be closed unexpectedly. Or something between the client and the server (a switch or router) dropped the connection.
It might be server code that caused the problem, and it might not be. If you have access to the server code, you can put some debugging in there to tell you when client connections are closed. That might give you some indication of when and why connections are being dropped.
On the client, you have to write your code to take into account the possibility of the server failing at any time. That's just the way it is: network connections are inherently unreliable.
I was sending the HttpWebRequest from Console App, and UserAgent was
null by (default), so setting UserAgent worked along with setting
SecurityProtocol.
Should set SecurityProtocol before creating HttpWebRequest.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("yourpostURL");
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36";
The webrequest user agent is null by default. Just google "block empty user agent" and you'll find a strong desire of many web server admins to do just that.
Sending my request with
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0";
fixed the issue.
I get that problem in the past. I'm using PostgreSQL and when I run my program, sometimes it connects and sometimes it throws an error like that.
When I experiment with my code, I put my Connection code at the very first line below the public Form. Here is an example:
BEFORE:
public Form1()
{
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
}
NOW:
public Form1()
{
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
}
I think that the program must read first the connection before doing anything, I don't know, correct me if I'm wrong. But according to my research, it's not a code problem - it was actually from the machine itself.
System.Net.ServicePointManager.Expect100Continue = false;
This issue sometime occurs due the reason of proxy server implemented on web server. To bypass the proxy server by putting this line before calling the send service.
We had a very similar issue whereby a client's website was trying to connect to our Web API service and getting that same message. This started happening completely out of the blue when there had been no code changes or Windows updates on the server where IIS was running.
In our case it turned out that the calling website was using a version of .Net that only supported TLS 1.0 and for some reason the server where our IIS was running stopped appeared to have stopped accepting TLS 1.0 calls. To diagnose that we had to explicitly enable TLS via the registry on the IIS's server and then restart that server. These are the reg keys:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
If that doesn't do it, you could also experiment with adding the entry for SSL 2.0:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
My answer to another question here has this powershell script that we used to add the entries:
NOTE: Enabling old security protocols is not a good idea, the right answer in our case was to get the client website to update it's code to use TLS 1.2, but the registry entries above can help diagnose the issue in the first place.
The reason this was happening to me was I had a recursive dependency in my DI provider. In my case I had:
services.AddScoped(provider => new CfDbContext(builder.Options));
services.AddScoped(provider => provider.GetService<CfDbContext>());
Fix was to just remove the second scoped service registration
services.AddScoped(provider => new CfDbContext(builder.Options));
Had a similar problem and was getting the following errors depending on what app I used and if we bypassed the firewall / load balancer or not:
HTTPS handshake to [blah] (for #136) failed.
System.IO.IOException Unable to read data from the transport
connection: An existing connection was forcibly closed by the remote
host
and
ReadResponse() failed: The server did not return a complete response for this request. Server returned 0 bytes.
The problem turned out to be that the SSL Server Certificate got missed and wasn't installed on a couple servers.
For me, It was an issue where in the IIS binding it had the IP address of the web server.
I changed it to use all unassigned IPs and my application started to work.
I experienced the error with python clr running mdx query to Microsoft analytic services using adomd
I solved it with help of Hans Vonn and here is the python version:
clr.AddReference("System.Net")
from System.Net import ServicePointManager, SecurityProtocolType
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls
I received this error simply because I was attempting to make an http connection to an https-only server. Changing the request protocol in the URI from http to https thus resolved it.
This is how I solved the issue:
int i = 0;
while (stream.DataAvailable == true)
{
bytes[i] = ((byte)stream.ReadByte());
i++;
}
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
Console.WriteLine("Received: {0}", data);
I had a Third Party application (Fiddler) running to try and see the requests being sent. Closing this application fixed it for me
If you have a https certificate on the domain, make sure you have the https binding to the domain name in IIS.
In IIS -> Select your domain -> Click on Bindings
Site Bindings Window opens up. Add a binding for https.
Try checking if you can establish handshake in the first place. I had this issue before when uploading a file and I only figured out that the issue was the nonexistent route when I removed the upload and checked if it can login given the parameters.
Another option would be to check the error code generated using try-catch block and first catching a WebException.
In my case, the error code was "SendFailure" because of certificate issue on HTTPS url, once I hit HTTP, that got resolved.
https://learn.microsoft.com/en-us/dotnet/api/system.net.webexceptionstatus?redirectedfrom=MSDN&view=netframework-4.8
This problem occurring when the Service is Unavailable within the proxy server. We can bypass the proxy server.
Before start, the service, apply this code line.
System.Net.ServicePointManager.Expect100Continue = false;
Further details
In my case I resolved this problem setting a correct API's url in my application.
It was an error connection between the application and API.

Categories