Socrata : An existing connection was forcibly closed by the remote host - c#

I created an application that periodically(every 1st Sunday of the month) send request to data.cms.gov to check for pecos registered physician. The code went well, however it stopped working a few days ago. I'm getting this response "Unable to read data from transport connection. An existing connection was forcibly closed by the remote host." Who encounter this before or can somebody help with this? I use the code below for my request
string end_point = "https://data.cms.gov/resource/qcn7-gc3g.json?$$app_token=myapp_token&npi=";
string cms_uri = end_point + npi;
System.Net.WebClient cms_wc = new System.Net.WebClient();
byte[] bResponse = cms_wc.DownloadData(cms_uri);
string cms_response = System.Text.Encoding.ASCII.GetString(bResponse);

As a security upgrade, we've disabled TLS 1.0 as an allowable SSL protocol. My guess is that that's what's triggering your disconnect.
https://support.socrata.com/hc/en-us/articles/235267087
You'll need to instruct .NET to to use TLS 1.1 or 1.2. You should be able to do that by adding the following before you create your client:
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
There's some more info in this issue on one of the C# .NET libraries for the SODA API.

Related

Fluent FTP Won't connect to FTP Server [duplicate]

This question already has an answer here:
"The handshake failed due to an unexpected packet format" FluentFTP error when connecting to SFTP server
(1 answer)
Closed 1 year ago.
I'm trying to connect to an FTP Server using Fluent FTP.
I have no issue using Filezilla when I use Port 22 (SFTP).
However I can't connect to it using Fluent FTP.
var client = new FtpClient(Constants.FtpHost, Constants.FtpUsername, Constants.FtpPassword)
{
// DataConnectionType = FtpDataConnectionType.PASV,
// EncryptionMode = FtpEncryptionMode.Implicit,
// SslProtocols = SslProtocols.Tls | SslProtocols.Tls11 | SslProtocols.Tls12 | SslProtocols.Tls13,
Port = 22
};
Console.WriteLine("Connecting...");
client.Connect();
Console.WriteLine("Connected");
I Get the following error: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
When I Uncomment DataConnectionType = FtpDataConnectionType.PASV Nothing changes.
When I Uncomment EncryptionMode = FtpEncryptionMode.Implicit I Get the following Error:
System.IO.IOException: Cannot determine the frame size or a corrupted frame was received.
Once again I have 0 issues with FileZilla or WinSCP.
Any help would be greatly appreciated.
From the official GitHub page:
SFTP is not supported as it is FTP over SSH, a completely different protocol (use SSH.NET for that)
You're trying to use an FTP library to connect to an SFTP server. They're two completely different protocols that have nothing in common, despite fulfulling the same goal. Use a proper SFTP library for that server.
Both FileZilla and WinSCP support both FTP and SFTP, that's why you have no problems with them.

ASP.NET application initially fails with: An existing connection was forcibly closed by the remote host

I apologize if I missed any details in this post, it's a bit bewildering and seems to be more of a bug in the OS than anything else.
We have an ASP.NET web application running on Windows Server 2019. The application attempts to make a web connection to another hosted application. We are trying to force it to use TLS1.2. When we do, the initial connection request ALWAYS fails after restarting IIS and then works fine from that point on.
I've installed Wireshark and can confirm that there is NO TCP connection attempt made on the wire whatsoever. Instead, we get an error from the application that the connection was forcibly closed.
System.IO.IOException: Unable to read data from the transport
connection: An existing connection was forcibly closed by the remote
host. ---> System.Net.Sockets.SocketException: An existing connection
was forcibly closed by the remote host at
System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset,
Int32 size)
The ASP.NET application is using .NET 4.5.2. The simplified code looks like so:
try {
HttpWebRequest hwrequest = (HttpWebRequest) System.Net.WebRequest.Create("https://api.domain.com/");
hwrequest.Accept = "*/*";
hwrequest.AllowAutoRedirect = true;
hwrequest.UserAgent = "http_requester/0.1";
hwrequest.Timeout = 900000;
hwrequest.Method = "POST";
hwrequest.KeepAlive = false;
hwrequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(delegate {
return true;
});
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();
byte[] postByteArray = encoding.GetBytes(postData);
hwrequest.ContentLength = postByteArray.Length;
System.IO.Stream postStream = hwrequest.GetRequestStream();
postStream.Write(postByteArray, 0, postByteArray.Length);
postStream.Close();
System.Net.HttpWebResponse hwresponse = (System.Net.HttpWebResponse) hwrequest.GetResponse();
if (hwresponse.StatusCode == System.Net.HttpStatusCode.OK) {
System.IO.Stream responseStream = hwresponse.GetResponseStream();
System.IO.StreamReader myStreamReader = new System.IO.StreamReader(responseStream);
responseData = myStreamReader.ReadToEnd();
}
hwresponse.Close();
}
catch(Exception e) {
Console.WriteLine(e.ToString());
responseData = "An error occurred: " + e.Message;
}
If we refresh the page, the connection attempt succeeds. But the initial request fails. If we comment out the line ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; then there is no problem, but the application uses TLSv1.
Is there something that you can think of that would cause the initial connection attempt to fail and then all subsequent requests to work until IIS is restarted? Seems like some type of bug in the OS rather than a coding issue.
This is a fully patched Windows Server 2019 almost fresh out of the box on Amazon AWS. There is no security software or extra things installed on it.
Maybe this generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application
The network link between the client and server is going down for some
reason
You have triggered a bug in the third-party application that caused
it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
But You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem. Without more specific information, it's unlikely that anyone here can really help you much.
Another way is Using TLS 1.2 might solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :

HttpClient https request fails using hostname, but works using IP address

I am new to web stuff in general. Not sure what the issue is here.
I have a machine with IIS running a ASP NET Core REST API on https.
I can confirm the GET is working via Google Chrome on my machine by doing either
https://test-machine/api/Example (works)
https://10.0.0.21/api/Example (invalid certificate, N.B. this is as expected as the certificate is for "test-machine")
I confirmed the POST action is working using Postman (again ignoring invalid certificate).
Everything is fine till I try and write a client application.
I have the following code.
var ip = Dns.GetHostAddresses("test-machine");
// ip contains the correct IP
using (var hc = new HttpClient())
{
hc.GetAsync(#"https://test-machine/api/Example").Wait();
}
This code fails. I get an System.AggregateException: 'One or more errors occurred.' with 4 exceptions stacked up on top of one another.
HttpRequestException: An error occurred while sending the request.
WebException: The underlying connection was closed: An unexpected error occurred on a send.
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
If I change the code to use the IP instead:
using (var hc = new HttpClient())
{
hc.GetAsync(#"https://10.0.0.21/api/Example").Wait();
}
Then the request works as expected I.e. as with Chrome, I get a certificate problem:
AuthenticationException: The remote certificate is invalid according to the validation procedure.
What am I doing wrong? The Dns is clearly able to get the correct IP, I've seen it in the IPAddress[] returned by GetHostAddresses. Why is the HttpClient not resolving the IP - or is it resolving the IP with another problem?
I have tried
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
and
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
but this makes no difference. Using the
ServicePointManager.ServerCertificateValidationCallback
Also doesn't work as it never reaches the point of calling the callback. It's failing before then.
I suspect the problem is that as I am not familiar with any of this stuff I have missed something important somewhere.
I should note also - this is a .net 4.5.2 Console app. I tried moving to 4.6.1 and this still fails.
I was looking completely in the wrong place. The problem lay in the IIS setup.
The site binding was for some reason explicitly set to 10.0.0.21:433. Switching this to "All Unassigned" - seems to have made this work.
I am probably out of my depth, and could be wrong, but I suspect the reason for this is that according to Wireshark, the response to the MDNS request to find the hostname IP, was returning the ipv6 address of the server machine, which of course isn't 10.0.0.21. Again, I could be wrong.
The reason I am so unsure is that chrome seems to have ploughed right through and found the correct thing when I put in https://test-machine/api/Example and I cannot explain why.

C# ClientWebSocket retry after connection error

Is there any way to know whether ClientWebSocket connection is closed/disconnected? I am using slack RTM api in C# and sometimes due to connectivity issues connection is closed from server, then I want to retry connection.
I am connecting to websocket server like this
var client = new ClientWebSocket();
client.ConnectAsync(url, token);

Unable to read data from the transport connection : An existing connection was forcibly closed by the remote host

I have a server app and sometimes, when the client tries to connect, I get the following error:
NOTE: the "couldn't get stream from client or login failed" is a text that's added by me in catch statement
and the line at which it stops ( sThread : line 96 ) is :
tcpClient = (TcpClient)client;
clientStream = tcpClient.GetStream();
sr = new StreamReader(clientStream);
sw = new StreamWriter(clientStream);
// line 96:
a = sr.ReadLine();
What may be causing this problem? Note that it doesn't happen all the time
I received this error when calling a web-service. The issue was also related to transport level security. I could call the web-service through a website project, but when reusing the same code in a test project I would get a WebException that contained this message. Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Edit
System.Net.ServicePointManager.SecurityProtocol - This property
selects the version of the Secure Sockets Layer (SSL) or Transport
Layer Security (TLS) protocol to use for new connections that use the
Secure Hypertext Transfer Protocol (HTTPS) scheme only; existing
connections are not changed.
I believe the SecurityProtocol configuration is important during the TLS handshake when selecting the protocol version.
TLS handshake - This protocol is used to exchange all the information required by both sides for the exchange of the actual application data by TLS.
ClientHello - A client sends a ClientHello message specifying the highest TLS protocol version it supports ...
ServerHello - The server responds with a ServerHello message, containing the chosen protocol version ... The chosen protocol version should be the highest that both the client and server support. For example, if the client supports TLS version 1.1 and the server supports version 1.2, version 1.1 should be selected; version 1.2 should not be selected.
This error usually means that the target machine is running, but the service that you're trying to connect to is not available. (Either it stopped, crashed, or is busy with another request.)
In English:
The connection to the machine (remote host/server/PC that the service runs at) was made but since the service was not available on that machine, the machine didn't know what to do with the request.
If the connection to the machine was not available, you'd see a different error. I forget what it is, but it's along the lines of "Service Unreachable" or "Unavailable".
Edit - added
It IS possible that this is being caused by a firewall blocking the port, but given that you say it's intermittent ("sometimes when the client tries to connect"), that's very unlikely. I didn't include that originally because I had ruled it out mentally before replying.
My specific case scenario was that the Azure app service had the minimum TLS version changed to 1.2
I don't know if that's the default from now on, but changing it back to 1.0 made it work.
You can access the setting inside "SSL Settings".
According to "Hans Vonn" replies.
Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
After adding Security protocol and working fine but I have to add before every API call which is not healthy. I just upgrade .net framework version at least 4.6 and working as expected do not require to adding before every API call.
Not sure which of the fixes in these blog posts helped, but one of them sorted this issue for me ...
http://briancaos.wordpress.com/2012/07/06/unable-to-read-data-from-the-transport-connection-the-connection-was-closed/
The trick that helped me was to quit using a WebRequest and use a HttpWebRequest instead. The HttpWebRequest allows me to play with 3 important settings:
and
http://briancaos.wordpress.com/2012/06/15/an-existing-connection-was-forcibly-closed-by-the-remote-host/
STEP 1: Disable KeepAlive
STEP 2: Set ProtocolVersion to Version10
STEP 3: Limiting the number of service points
For those who may find this later, after .NET version 4.6, I was running into this problem as well.
Make sure that you check your web.config file for the following lines:
<compilation debug="true" targetFramework="4.5">
...
<httpRuntime targetFramework="4.5" />
If you are running 4.6.x or a higher version of .NET on the server, make sure you adjust these targetFramework values to match the version of the framework on your server. If your versions read less than 4.6.x, then I would recommend you upgrade .NET and use the newer version unless your code is dependent on an older version (which, in that case, you should consider updating it).
I changed the targetFrameworks to 4.7.2 and the problem disappeared:
<compilation debug="true" targetFramework="4.7.2">
...
<httpRuntime targetFramework="4.7.2" />
The newer frameworks sort this issue out by using the best protocol available and blocking insecure or obsolete ones. If the remote service you are trying to connect to or call is giving this error, it could be that they don't support the old protocols anymore.
Calls to HTTPS services from one of our servers were also throwing the "Unable to read data from the transport connection : An existing connection was forcibly closed" exception. HTTP service, though, worked fine. Used Wireshark to see that it was a TLS handshake Failure. Ended up being that the cipher suite on the server needed to be updated.
This solved my problem. I added this line before the request is made:
System.Net.ServicePointManager.Expect100Continue = false;
It seemed there were a proxy in the way of the server that not supported 100-continue behavior.
This won't help for intermittent issues, but may be useful for other people with a similar problem.
I had cloned a VM and started it up on a different network with a new IP address but not changed the bindings in IIS. Fiddler was showing me "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host" and IE was telling me "Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings". Changing the binding to the new IP address solved it for me.
For some reason, the connection to the server was lost. It could be that the server explicitly closed the connection, or a bug on the server caused it to be closed unexpectedly. Or something between the client and the server (a switch or router) dropped the connection.
It might be server code that caused the problem, and it might not be. If you have access to the server code, you can put some debugging in there to tell you when client connections are closed. That might give you some indication of when and why connections are being dropped.
On the client, you have to write your code to take into account the possibility of the server failing at any time. That's just the way it is: network connections are inherently unreliable.
I was sending the HttpWebRequest from Console App, and UserAgent was
null by (default), so setting UserAgent worked along with setting
SecurityProtocol.
Should set SecurityProtocol before creating HttpWebRequest.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("yourpostURL");
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36";
The webrequest user agent is null by default. Just google "block empty user agent" and you'll find a strong desire of many web server admins to do just that.
Sending my request with
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0";
fixed the issue.
I get that problem in the past. I'm using PostgreSQL and when I run my program, sometimes it connects and sometimes it throws an error like that.
When I experiment with my code, I put my Connection code at the very first line below the public Form. Here is an example:
BEFORE:
public Form1()
{
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
}
NOW:
public Form1()
{
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
}
I think that the program must read first the connection before doing anything, I don't know, correct me if I'm wrong. But according to my research, it's not a code problem - it was actually from the machine itself.
System.Net.ServicePointManager.Expect100Continue = false;
This issue sometime occurs due the reason of proxy server implemented on web server. To bypass the proxy server by putting this line before calling the send service.
We had a very similar issue whereby a client's website was trying to connect to our Web API service and getting that same message. This started happening completely out of the blue when there had been no code changes or Windows updates on the server where IIS was running.
In our case it turned out that the calling website was using a version of .Net that only supported TLS 1.0 and for some reason the server where our IIS was running stopped appeared to have stopped accepting TLS 1.0 calls. To diagnose that we had to explicitly enable TLS via the registry on the IIS's server and then restart that server. These are the reg keys:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
If that doesn't do it, you could also experiment with adding the entry for SSL 2.0:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
My answer to another question here has this powershell script that we used to add the entries:
NOTE: Enabling old security protocols is not a good idea, the right answer in our case was to get the client website to update it's code to use TLS 1.2, but the registry entries above can help diagnose the issue in the first place.
The reason this was happening to me was I had a recursive dependency in my DI provider. In my case I had:
services.AddScoped(provider => new CfDbContext(builder.Options));
services.AddScoped(provider => provider.GetService<CfDbContext>());
Fix was to just remove the second scoped service registration
services.AddScoped(provider => new CfDbContext(builder.Options));
Had a similar problem and was getting the following errors depending on what app I used and if we bypassed the firewall / load balancer or not:
HTTPS handshake to [blah] (for #136) failed.
System.IO.IOException Unable to read data from the transport
connection: An existing connection was forcibly closed by the remote
host
and
ReadResponse() failed: The server did not return a complete response for this request. Server returned 0 bytes.
The problem turned out to be that the SSL Server Certificate got missed and wasn't installed on a couple servers.
For me, It was an issue where in the IIS binding it had the IP address of the web server.
I changed it to use all unassigned IPs and my application started to work.
I experienced the error with python clr running mdx query to Microsoft analytic services using adomd
I solved it with help of Hans Vonn and here is the python version:
clr.AddReference("System.Net")
from System.Net import ServicePointManager, SecurityProtocolType
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls
I received this error simply because I was attempting to make an http connection to an https-only server. Changing the request protocol in the URI from http to https thus resolved it.
This is how I solved the issue:
int i = 0;
while (stream.DataAvailable == true)
{
bytes[i] = ((byte)stream.ReadByte());
i++;
}
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
Console.WriteLine("Received: {0}", data);
I had a Third Party application (Fiddler) running to try and see the requests being sent. Closing this application fixed it for me
If you have a https certificate on the domain, make sure you have the https binding to the domain name in IIS.
In IIS -> Select your domain -> Click on Bindings
Site Bindings Window opens up. Add a binding for https.
Try checking if you can establish handshake in the first place. I had this issue before when uploading a file and I only figured out that the issue was the nonexistent route when I removed the upload and checked if it can login given the parameters.
Another option would be to check the error code generated using try-catch block and first catching a WebException.
In my case, the error code was "SendFailure" because of certificate issue on HTTPS url, once I hit HTTP, that got resolved.
https://learn.microsoft.com/en-us/dotnet/api/system.net.webexceptionstatus?redirectedfrom=MSDN&view=netframework-4.8
This problem occurring when the Service is Unavailable within the proxy server. We can bypass the proxy server.
Before start, the service, apply this code line.
System.Net.ServicePointManager.Expect100Continue = false;
Further details
In my case I resolved this problem setting a correct API's url in my application.
It was an error connection between the application and API.

Categories