gRPC connection gets cut after 60 seconds if inactivity - c#

I have been trying to set up a gRPC API capable of streaming events to a client. Basically, after a client has subscribed, the server will use gRPC's "Server Streaming" feature to send any new event to the client.
I expect there to be periods of inactivity, where the connection should remain active. However, with my current setup it seems Nginx is cutting the connection after 60 seconds of inactivity with the following exception at the client:
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely, with at least 9 additional bytes expected.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
The question is why? and how can I prevent it?
My setup
The API is built in ASP.NET Core 3 (will probably upgrade to .NET 5 soon) and is running in a Docker container on a Digital Ocean server.
Nginx is also running in a Docker container on the server and works as a reverse proxy for the API (among other things).
The client is a simple C# client written in .NET Core and is run locally.
What have I tried?
I have tried to connect to the Docker image directly on the server using grpc_cli (bypassing Nginx) where the connection remain active for long periods of inactivity without any issues. So I can't see what else it can be, except Nginx. Also, most of Nginx default timeout values seem to be 60 seconds.
I have tried these Nginx settings and various combinations of them, yet haven't found the right one (or the right combination) yet:
location /commands.CommandService/ {
grpc_pass grpc://commandApi;
grpc_socket_keepalive on;
grpc_read_timeout 3000s; # These are recommended everywhere, but I haven't had any success
grpc_send_timeout 3000s; #
grpc_next_upstream_timeout 0;
proxy_request_buffering off;
proxy_buffering off;
proxy_connect_timeout 3000s;
proxy_send_timeout 3000s;
proxy_read_timeout 3000s;
proxy_socket_keepalive on;
keepalive_timeout 90s;
send_timeout 90s;
client_body_timeout 3000s;
}
The most common suggestion for people with similar issues is to use grpc_read_timeout and grpc_send_timeout, but they don't work for me. I guess it makes sense since I'm not actively sending/receiving anything.
My client code looks like this:
var httpClientHandler = new HttpClientHandler();
var channel = GrpcChannel.ForAddress("https://myapi.com", new GrpcChannelOptions()
{
HttpClient = new HttpClient(httpClientHandler) { Timeout = Timeout.InfiniteTimeSpan },
});
var commandService = channel.CreateGrpcService<ICommandService>();
var request = new CommandSubscriptionRequest()
{
HandlerId = _handlerId
};
var sd = new CancellationTokenSource();
var r = new CallContext(callOptions: new CallOptions(deadline: null, cancellationToken: sd.Token));
await foreach (var command in commandService.SubscribeCommandsAsync(request, r))
{
Console.WriteLine("Processing command: " + command.Id);
}
return channel;
To be clear, the call to the API works and I can receive commands from the server. If I just keep sending commands from the API, everything is working beautifully. But as soon as I stop for 60 seconds (I have timed it), the connection breaks.
A possible workaround would be to just keep sending a kind of heartbeat to keep the connection open, but I would prefer not to.
Does anyone know how I can fix it? Am I missing something obvious?
UPDATE: Turns out it wasn't Nginx. After I updated the API and the client to .NET 5 the problem disappeared. I can't say in what version this was fixed, but at least it's gone in .NET 5.

Not sure this is an Nginx issue, looks like a client connection problem.
Your results look very similar to an issue I had, that should have been fixed in .net 3.0 patch. Try updating to a newer version of .NET and see if that fixes the problem.
Alternatively, it could be a problem with the max number of connections. Try setting the MaxConcurrentConnections for the kestrel server (in appsettings.json):
{
"Kestrel": {
"Limits": {
"MaxConcurrentConnections": 100,
"MaxConcurrentUpgradedConnections": 100
}
}
}

Related

.NET: How to send TCP Keep Alive packets while downloading from HTTP API?

With a .NET/C# application I am trying to download data from an HTTP API. Even though the timeout of the HttpClient instance is set to 30 minutes, the request will time out much faster. Today I learned this is due to the fact that the .NET HttpClient does not send any TCP Keep Alive packets. (This is why I can download data from that API in Chrome, as proven with Wireshark, Chrome does send these packets while HttpClient does not.)
This is how I had hoped to get the JSON data from the API:
this.httpclient = new HttpClient();
[...]
result = await this.httpclient.GetAsync(url);
Now I did some research but I could not find out how to send these Keep Alive pings. The HttpClient does not seem to support them. So my guess is these need to be enabled on the socket when opening the connection. Is there any workaround for this, maybe avoiding HttpClient? Can I enable the Keep-Alive pings somehow on the socket? Is this a feature that needs to be enabled through the OS?
Please note: This is NOT about the HTTP "Connection: Keep-Alive" header! You can enable this for the HttpClient class but for obvious reasons it won't trigger any TCP Keep Alive packets.
Edit:
The requests always work when these Keep-Alive packets are being sent by the client, they will fail if not. This can be reproduced by using different clients/browsers. Best example: The Postman Chrome extension does send Keep-Alive packets, hence the request works. Using the almost identical Postman standalone client the request fails (or does not return any data) because, funnily enough, the standalone client does NOT send any Keep-Alive packets which can easily be checked with Wireshark.
Edit: I have found a very simple solution for this problem, see my answer below.
So now I end up answering my own question as I just have found a super simple solution that actually works:
this.httpclient = new HttpClient();
var sp = ServicePointManager.FindServicePoint(new Uri(url));
sp.SetTcpKeepAlive(true, 30000, 30000);
result = await this.httpclient.GetAsync(url);
In fact I used
var sp = ServicePointManager.FindServicePoint(new Uri(baseUrl));
where baseUrl is the common document root of all my requests I am sending through a for loop.
HTTP (unlike WebSockets) doesn't support keepalive messages, so I assume you're referring to SO_KEEPALIVE-style TCP keepalive packets.
Even though the timeout of the HttpClient instance is set to 30 minutes, the request will time out much faster. Today I learned this is due to the fact that the .NET HttpClient does not send any TCP Keep Alive packets.
I don't think that's correct. The sending of keepalive packets wouldn't have any effect on timeout behavior. In particular, when the keepalive reaches the server and it sends the ack response, neither application is even notified - indeed they cannot be notified since there is no data in a TCP keepalive or ack.
Now I did some research but I could not find out how to send these Keep Alive pings.
TCP keepalive packets are problematic for two reasons: they have awkward defaults, and they can be dropped by intermediate routers. The awkward defaults include a minimum timer of 2 hours and no guarantee of the ability to change that; fortunately, modern Windows versions do allow setting a per-connection keepalive timer, and allow setting it to a much lower value.
That said, I don't know of a way to get to the underlying Socket for an HttpClient. The HttpClient has a pipeline of handlers which end (these days) in a SocketsHttpHandler, which is actually responsible for the socket connection. But I don't see any APIs on that type that let you reach directly in to manipulate the socket, or provide an already-configured socket during construction.
Here is another solution from NetCore 2.1 onwards.I found the ServicePointManager.FindServicePoint(new Uri(url)); code did not work in my case, this did.
Source https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.connectcallback?view=net-7.0#examples
using SocketsHttpHandler handler = new SocketsHttpHandler();
handler.ConnectCallback = async (ctx, ct) =>
{
var s = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
try
{
s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
s.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, 5);
s.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, 5);
s.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, 5); //note this doesnt work on some windows versions
await s.ConnectAsync(ctx.DnsEndPoint, ct);
return new NetworkStream(s, ownsSocket: true);
}
catch
{
s.Dispose();
throw;
}
};
// Create an HttpClient object
using HttpClient client = new HttpClient(handler);

Weird WCF behaviour regarding Timouts and Exceptions

I have created an WCF service hosted inside a normal Windows service. This service is deployed to customers and set up on their servers. Therefore (afaik) I need to establish the WCF proxy dynamically and cannot rely on some prebuilt proxy created by VS or the Silverlight tools. The clients in this case are mobile apps built with Xamarin.Forms.
The Code to create the "Channel":
public void Init(int timeout = 15)
{
ea = new EndpointAddress(string.Format("http://{0}:{1}/{2}", _settingsService.ConnectionIP, _settingsService.ConnectionPort, _settingsService.ConnectionEndpoint));
bhttpb = new BasicHttpBinding(BasicHttpSecurityMode.None);
bhttpb.SendTimeout = TimeSpan.FromSeconds(timeout);
cfIMMC = new ChannelFactory<IMaintMobileContract>(bhttpb, ea);
cfIMMC.Opened += cfIMMC_Opened;
cfIMMC.Faulted += cfIMMC_Faulted;
cfIMMC.Closed += cfIMMC_Closed;
immc = cfIMMC.CreateChannel(ea);
immc.Ping(); // This function is defined by me in the Contract. It only returns true, if the server can be reached.
}
So far everything works fine if the service is running, but the app has to run "offline" and then it gets weird.
When the connection is established there is no EndpointException or anything, and when a function is called it just sits there waiting until the timeout hits.
It would be really nice to get some information whether the WCF service is actually there or not. I have function calls that can take up to multiple minutes and it would be fatal for the app to wait that long when the WCF server is not there at all. How can I achieve that?
Update:
Right now it got even weirder. Now, aprox. 30 seconds after the Ping() fails, I get System.Net.Sockets.SocketException: Connection timed out and System.Net.WebException: Error: ConnectFailure (Connection timed out) out of nowhere.
Update 2 :
Here a pic of the CallStack:
If you need fast feedback regarding whether service is alive or not, then setup additional endpoint (with separate contract containing only Ping method) and set small timeouts for it.
And important part is to set send/receive timeouts to small value as well - this will ensure that Ping method returns/throws fast if service is not available.
As far as I remember WCF does not open channel (== does not connect to server) until you call one of the methods - that's why you don't have exceptions before Ping is called.
About exception after 30 seconds. Where do you see it? I mean is it Visual Studio that breaks there or do you have your application failing with unhandled exception? I'm asking it because I see this in the Xamarin/Mono code:
initConn = new WaitCallback (state => {
try {
InitConnection (state);
} catch {}
});
And it means that even though this exception is thrown after 30 seconds - it'll be swallowed. What really happens is that when request is sent (i.e. when you call Ping()) the runtime tries to open connection in background (your call stack confirms that) and 30 seconds is default Windows timeout for connection. WCF will fail earlier if it has lower timeout set (like in your case), but connection attempt will last for 30 seconds and will complete with exception.
So, my opinion is that you should not care about this exception, unless it somehow stops your application.

Unable to read data from the transport connection : An existing connection was forcibly closed by the remote host

I have a server app and sometimes, when the client tries to connect, I get the following error:
NOTE: the "couldn't get stream from client or login failed" is a text that's added by me in catch statement
and the line at which it stops ( sThread : line 96 ) is :
tcpClient = (TcpClient)client;
clientStream = tcpClient.GetStream();
sr = new StreamReader(clientStream);
sw = new StreamWriter(clientStream);
// line 96:
a = sr.ReadLine();
What may be causing this problem? Note that it doesn't happen all the time
I received this error when calling a web-service. The issue was also related to transport level security. I could call the web-service through a website project, but when reusing the same code in a test project I would get a WebException that contained this message. Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
Edit
System.Net.ServicePointManager.SecurityProtocol - This property
selects the version of the Secure Sockets Layer (SSL) or Transport
Layer Security (TLS) protocol to use for new connections that use the
Secure Hypertext Transfer Protocol (HTTPS) scheme only; existing
connections are not changed.
I believe the SecurityProtocol configuration is important during the TLS handshake when selecting the protocol version.
TLS handshake - This protocol is used to exchange all the information required by both sides for the exchange of the actual application data by TLS.
ClientHello - A client sends a ClientHello message specifying the highest TLS protocol version it supports ...
ServerHello - The server responds with a ServerHello message, containing the chosen protocol version ... The chosen protocol version should be the highest that both the client and server support. For example, if the client supports TLS version 1.1 and the server supports version 1.2, version 1.1 should be selected; version 1.2 should not be selected.
This error usually means that the target machine is running, but the service that you're trying to connect to is not available. (Either it stopped, crashed, or is busy with another request.)
In English:
The connection to the machine (remote host/server/PC that the service runs at) was made but since the service was not available on that machine, the machine didn't know what to do with the request.
If the connection to the machine was not available, you'd see a different error. I forget what it is, but it's along the lines of "Service Unreachable" or "Unavailable".
Edit - added
It IS possible that this is being caused by a firewall blocking the port, but given that you say it's intermittent ("sometimes when the client tries to connect"), that's very unlikely. I didn't include that originally because I had ruled it out mentally before replying.
My specific case scenario was that the Azure app service had the minimum TLS version changed to 1.2
I don't know if that's the default from now on, but changing it back to 1.0 made it work.
You can access the setting inside "SSL Settings".
According to "Hans Vonn" replies.
Adding the following line before making the call resolved the issue:
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;
After adding Security protocol and working fine but I have to add before every API call which is not healthy. I just upgrade .net framework version at least 4.6 and working as expected do not require to adding before every API call.
Not sure which of the fixes in these blog posts helped, but one of them sorted this issue for me ...
http://briancaos.wordpress.com/2012/07/06/unable-to-read-data-from-the-transport-connection-the-connection-was-closed/
The trick that helped me was to quit using a WebRequest and use a HttpWebRequest instead. The HttpWebRequest allows me to play with 3 important settings:
and
http://briancaos.wordpress.com/2012/06/15/an-existing-connection-was-forcibly-closed-by-the-remote-host/
STEP 1: Disable KeepAlive
STEP 2: Set ProtocolVersion to Version10
STEP 3: Limiting the number of service points
For those who may find this later, after .NET version 4.6, I was running into this problem as well.
Make sure that you check your web.config file for the following lines:
<compilation debug="true" targetFramework="4.5">
...
<httpRuntime targetFramework="4.5" />
If you are running 4.6.x or a higher version of .NET on the server, make sure you adjust these targetFramework values to match the version of the framework on your server. If your versions read less than 4.6.x, then I would recommend you upgrade .NET and use the newer version unless your code is dependent on an older version (which, in that case, you should consider updating it).
I changed the targetFrameworks to 4.7.2 and the problem disappeared:
<compilation debug="true" targetFramework="4.7.2">
...
<httpRuntime targetFramework="4.7.2" />
The newer frameworks sort this issue out by using the best protocol available and blocking insecure or obsolete ones. If the remote service you are trying to connect to or call is giving this error, it could be that they don't support the old protocols anymore.
Calls to HTTPS services from one of our servers were also throwing the "Unable to read data from the transport connection : An existing connection was forcibly closed" exception. HTTP service, though, worked fine. Used Wireshark to see that it was a TLS handshake Failure. Ended up being that the cipher suite on the server needed to be updated.
This solved my problem. I added this line before the request is made:
System.Net.ServicePointManager.Expect100Continue = false;
It seemed there were a proxy in the way of the server that not supported 100-continue behavior.
This won't help for intermittent issues, but may be useful for other people with a similar problem.
I had cloned a VM and started it up on a different network with a new IP address but not changed the bindings in IIS. Fiddler was showing me "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host" and IE was telling me "Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings". Changing the binding to the new IP address solved it for me.
For some reason, the connection to the server was lost. It could be that the server explicitly closed the connection, or a bug on the server caused it to be closed unexpectedly. Or something between the client and the server (a switch or router) dropped the connection.
It might be server code that caused the problem, and it might not be. If you have access to the server code, you can put some debugging in there to tell you when client connections are closed. That might give you some indication of when and why connections are being dropped.
On the client, you have to write your code to take into account the possibility of the server failing at any time. That's just the way it is: network connections are inherently unreliable.
I was sending the HttpWebRequest from Console App, and UserAgent was
null by (default), so setting UserAgent worked along with setting
SecurityProtocol.
Should set SecurityProtocol before creating HttpWebRequest.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("yourpostURL");
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36";
The webrequest user agent is null by default. Just google "block empty user agent" and you'll find a strong desire of many web server admins to do just that.
Sending my request with
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0";
fixed the issue.
I get that problem in the past. I'm using PostgreSQL and when I run my program, sometimes it connects and sometimes it throws an error like that.
When I experiment with my code, I put my Connection code at the very first line below the public Form. Here is an example:
BEFORE:
public Form1()
{
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
}
NOW:
public Form1()
{
//Connect to Database to generate auto number
NpgsqlConnection iConnect = new NpgsqlConnection("Server=localhost;Port=5432;User ID=postgres;Password=pass;Database=DB");
iConnect.Open();
NpgsqlCommand iQuery = new NpgsqlCommand("Select * from table1", iConnect);
NpgsqlDataReader iRead = iQuery.ExecuteReader();
NpgsqlDataAdapter iAdapter = new NpgsqlDataAdapter(iQuery);
DataSet iDataSet = new DataSet();
iAdapter.Fill(iDataSet, "ID");
MessageBox.Show(iDataSet.Tables["ID"].Rows.Count.ToString());
//HERE LIES SOME CODES FOR RESIZING MY CONTROLS DURING RUNTIME
//CODE
//CODE AGAIN
//ANOTHER CODE
//CODE NA NAMAN
//CODE PA RIN!
}
I think that the program must read first the connection before doing anything, I don't know, correct me if I'm wrong. But according to my research, it's not a code problem - it was actually from the machine itself.
System.Net.ServicePointManager.Expect100Continue = false;
This issue sometime occurs due the reason of proxy server implemented on web server. To bypass the proxy server by putting this line before calling the send service.
We had a very similar issue whereby a client's website was trying to connect to our Web API service and getting that same message. This started happening completely out of the blue when there had been no code changes or Windows updates on the server where IIS was running.
In our case it turned out that the calling website was using a version of .Net that only supported TLS 1.0 and for some reason the server where our IIS was running stopped appeared to have stopped accepting TLS 1.0 calls. To diagnose that we had to explicitly enable TLS via the registry on the IIS's server and then restart that server. These are the reg keys:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.0\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.1\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS
1.2\Server] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001
If that doesn't do it, you could also experiment with adding the entry for SSL 2.0:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
My answer to another question here has this powershell script that we used to add the entries:
NOTE: Enabling old security protocols is not a good idea, the right answer in our case was to get the client website to update it's code to use TLS 1.2, but the registry entries above can help diagnose the issue in the first place.
The reason this was happening to me was I had a recursive dependency in my DI provider. In my case I had:
services.AddScoped(provider => new CfDbContext(builder.Options));
services.AddScoped(provider => provider.GetService<CfDbContext>());
Fix was to just remove the second scoped service registration
services.AddScoped(provider => new CfDbContext(builder.Options));
Had a similar problem and was getting the following errors depending on what app I used and if we bypassed the firewall / load balancer or not:
HTTPS handshake to [blah] (for #136) failed.
System.IO.IOException Unable to read data from the transport
connection: An existing connection was forcibly closed by the remote
host
and
ReadResponse() failed: The server did not return a complete response for this request. Server returned 0 bytes.
The problem turned out to be that the SSL Server Certificate got missed and wasn't installed on a couple servers.
For me, It was an issue where in the IIS binding it had the IP address of the web server.
I changed it to use all unassigned IPs and my application started to work.
I experienced the error with python clr running mdx query to Microsoft analytic services using adomd
I solved it with help of Hans Vonn and here is the python version:
clr.AddReference("System.Net")
from System.Net import ServicePointManager, SecurityProtocolType
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls
I received this error simply because I was attempting to make an http connection to an https-only server. Changing the request protocol in the URI from http to https thus resolved it.
This is how I solved the issue:
int i = 0;
while (stream.DataAvailable == true)
{
bytes[i] = ((byte)stream.ReadByte());
i++;
}
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
Console.WriteLine("Received: {0}", data);
I had a Third Party application (Fiddler) running to try and see the requests being sent. Closing this application fixed it for me
If you have a https certificate on the domain, make sure you have the https binding to the domain name in IIS.
In IIS -> Select your domain -> Click on Bindings
Site Bindings Window opens up. Add a binding for https.
Try checking if you can establish handshake in the first place. I had this issue before when uploading a file and I only figured out that the issue was the nonexistent route when I removed the upload and checked if it can login given the parameters.
Another option would be to check the error code generated using try-catch block and first catching a WebException.
In my case, the error code was "SendFailure" because of certificate issue on HTTPS url, once I hit HTTP, that got resolved.
https://learn.microsoft.com/en-us/dotnet/api/system.net.webexceptionstatus?redirectedfrom=MSDN&view=netframework-4.8
This problem occurring when the Service is Unavailable within the proxy server. We can bypass the proxy server.
Before start, the service, apply this code line.
System.Net.ServicePointManager.Expect100Continue = false;
Further details
In my case I resolved this problem setting a correct API's url in my application.
It was an error connection between the application and API.

Enyim Memcached Client does not write / read data

I've installed memcached on Windows as a service, listening on the default port 11211. I know this works, because I can telnet to the server and carry out get / set commands without any problems.
I've then downloaded the Enyim Memcached client (Enyim.Caching.dll, version 2.7) and written a simple test program:
var mcc = new MemcachedClientConfiguration();
mcc.AddServer("127.0.0.1:11211");
mcc.SocketPool.ReceiveTimeout = new TimeSpan(0, 0, 10);
mcc.SocketPool.ConnectionTimeout = new TimeSpan(0, 0, 10);
mcc.SocketPool.DeadTimeout = new TimeSpan(0, 0, 20);
using (MemcachedClient client = new MemcachedClient(mcc))
{
client.Store(StoreMode.Set, "enyimtest", "test value");
Console.WriteLine(client.Get<string>("enyimtest"));
}
I know this connects to my server correctly, as calling the stats command in telnet shows an increase in the number of connections. However, it doesn't call get or set, as the cmd_get and cmd_set stats counters remain constant. The call to client.Get returns null.
The program does not error in any way. Does anyone know what could prevent the Enyim client from working in this situation?
EDIT:
Looks like this is caused by a timeout. Afer configuring log4net to capture the client's logging output, I found it contained the following (in addition to other stack trace items):
2010-12-17 14:26:37,579 [1] ERROR Enyim.Caching.Memcached.MemcachedNode [(null)] - System.IO.IOException: Failed to read from the socket '172.23.0.100:11211'. Error: TimedOut
2010-12-17 14:26:37,626 [1] WARN Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] - Marking node 172.23.0.100:11211 as dead
I still don't understand why it is timing out though?
After an hour or so of playing around, I've found the answer. I used Wireshark to look at the network traffic to and from the server. I noticed that when using the Enyim client, the messages looked nothing like those when using telnet. In particular, I couldn't read the protocol commands going across the wire when using the Enyim client.
Therefore, I concluded that the Enyim client was using a different protocol.
A second protocol was added to the memcached server in version 1.4, which is the binary protocol. Prior to that, only the text protocol was supported. The latest Windows binary I can find for memcached is the one from Jellycan, and it is only version 1.2.6.
The Enyim client is configured to use the Binary protocol by default, which was just ignored by my server as it couldn't be understood.
I added the following line to my test program, and things started working immediately:
mcc.Protocol = MemcachedProtocol.Text;
I ran into the same issue above. I too struggled to find a newer version of memcached for Windows, but did find one eventually.
I've put links to the latest binaries along with other useful resources here.

An existing connection was forcibly closed by the remote host

I am working with a commercial application which is throwing a SocketException with the message,
An existing connection was forcibly closed by the remote host
This happens with a socket connection between client and server. The connection is alive and well, and heaps of data is being transferred, but it then becomes disconnected out of nowhere.
Has anybody seen this before? What could the causes be? I can kind of guess a few causes, but also is there any way to add more into this code to work out what the cause could be?
Any comments / ideas are welcome.
... The latest ...
I have some logging from some .NET tracing,
System.Net.Sockets Verbose: 0 : [8188] Socket#30180123::Send() DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Error: 0 : [8188] Exception in the Socket#30180123::Send - An existing connection was forcibly closed by the remote host DateTime=2010-04-07T20:49:48.6317500Z
System.Net.Sockets Verbose: 0 : [8188] Exiting Socket#30180123::Send() -> 0#0
Based on other parts of the logging I have seen the fact that it says 0#0 means a packet of 0 bytes length is being sent. But what does that really mean?
One of two possibilities is occurring, and I am not sure which,
The connection is being closed, but data is then being written to the socket, thus creating the exception above. The 0#0 simply means that nothing was sent because the socket was already closed.
The connection is still open, and a packet of zero bytes is being sent (i.e. the code has a bug) and the 0#0 means that a packet of zero bytes is trying to be sent.
What do you reckon? It might be inconclusive I guess, but perhaps someone else has seen this kind of thing?
This generally means that the remote side closed the connection (usually by sending a TCP/IP RST packet). If you're working with a third-party application, the likely causes are:
You are sending malformed data to the application (which could include sending an HTTPS request to an HTTP server)
The network link between the client and server is going down for some reason
You have triggered a bug in the third-party application that caused it to crash
The third-party application has exhausted system resources
It's likely that the first case is what's happening.
You can fire up Wireshark to see exactly what is happening on the wire to narrow down the problem.
Without more specific information, it's unlikely that anyone here can really help you much.
Using TLS 1.2 solved this error.
You can force your application using TLS 1.2 with this (make sure to execute it before calling your service):
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Another solution :
Enable strong cryptography in your local machine or server in order to use TLS1.2 because by default it is disabled so only TLS1.0 is used.
To enable strong cryptography , execute these commande in PowerShell with admin privileges :
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type DWord
You need to reboot your computer for these changes to take effect.
This is not a bug in your code. It is coming from .Net's Socket implementation. If you use the overloaded implementation of EndReceive as below you will not get this exception.
SocketError errorCode;
int nBytesRec = socket.EndReceive(ar, out errorCode);
if (errorCode != SocketError.Success)
{
nBytesRec = 0;
}
Had the same bug. Actually worked in case the traffic was sent using some proxy (fiddler in my case). Updated .NET framework from 4.5.2 to >=4.6 and now everything works fine. The actual request was:
new WebClient().DownloadData("URL");
The exception was:
SocketException: An existing connection was forcibly closed by the
remote host
Simple solution for this common annoying issue:
Just go to your ".context.cs" file (located under ".context.tt" which located under your "*.edmx" file).
Then, add this line to your constructor:
public DBEntities()
: base("name=DBEntities")
{
this.Configuration.ProxyCreationEnabled = false; // ADD THIS LINE!
}
I've got this exception because of circular reference in entity.In entity that look like
public class Catalog
{
public int Id { get; set; }
public int ParentId { get; set; }
public Catalog Parent { get; set; }
public ICollection<Catalog> ChildCatalogs { get; set; }
}
I added [IgnoreDataMemberAttribute] to the Parent property. And that solved the problem.
If Running In A .Net 4.5.2 Service
For me the issue was compounded because the call was running in a .Net 4.5.2 service. I followed #willmaz suggestion but got a new error.
In running the service with logging turned on, I viewed the handshaking with the target site would initiate ok (and send the bearer token) but on the following step to process the Post call, it would seem to drop the auth token and the site would reply with Unauthorized.
Solution
It turned out that the service pool credentials did not have rights to change TLS (?) and when I put in my local admin account into the pool, it all worked.
I had the same issue and managed to resolve it eventually. In my case, the port that the client sends the request to did not have a SSL cert binding to it. So I fixed the issue by binding a SSL cert to the port on the server side. Once that was done, this exception went away.
For anyone getting this exception while reading data from the stream, this may help. I was getting this exception when reading the HttpResponseMessage in a loop like this:
using (var remoteStream = await response.Content.ReadAsStreamAsync())
using (var content = File.Create(DownloadPath))
{
var buffer = new byte[1024];
int read;
while ((read = await remoteStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await content.WriteAsync(buffer, 0, read);
await content.FlushAsync();
}
}
After some time I found out the culprit was the buffer size, which was too small and didn't play well with my weak Azure instance. What helped was to change the code to:
using (Stream remoteStream = await response.Content.ReadAsStreamAsync())
using (FileStream content = File.Create(DownloadPath))
{
await remoteStream.CopyToAsync(content);
}
CopyTo() method has a default buffer size of 81920. The bigger buffer sped up the process and the errors stopped immediately, most likely because the overall download speeds increased. But why would download speed matter in preventing this error?
It is possible that you get disconnected from the server because the download speeds drop below minimum threshold the server is configured to allow. For example, in case the application you are downloading the file from is hosted on IIS, it can be a problem with http.sys configuration:
"Http.sys is the http protocol stack that IIS uses to perform http communication with clients. It has a timer called MinBytesPerSecond that is responsible for killing a connection if its transfer rate drops below some kb/sec threshold. By default, that threshold is set to 240 kb/sec."
The issue is described in this old blogpost from TFS development team and concerns IIS specifically, but may point you in a right direction. It also mentions an old bug related to this http.sys attribute: link
In case you are using Azure app services and increasing the buffer size does not eliminate the problem, try to scale up your machine as well. You will be allocated more resources including connection bandwidth.
I got the same issue while using .NET Framework 4.5. However, when I update the .NET version to 4.7.2 connection issue was resolved. Maybe this is due to SecurityProtocol support issue.
For me, it was because the app server I was trying to send email from was not added to our company's SMTP server's allowed list.
I just had to put in SMTP access request for that app server.
This is how it was added by the infrastructure team (I don't know how to do these steps myself but this is what they said they did):
1. Log into active L.B.
2. Select: Local Traffic > iRules > Data Group List
3. Select the appropriate Data Group
4. Enter the app server's IP address
5. Select: Add
6. Select: Update
7. Sync config changes
Yet another possibility for this error to occur is if you tried to connect to a third-party server with invalid credentials too many times and a system like Fail2ban is blocking your IP address.
I was trying to connect to the MQTT broker using the GO client,
broker address was given as address + port, or tcp://address:port
Example: ❌
mqtt://test.mosquitto.org
which indicates that you wish to establish an unencrypted connection.
To request MQTT over TLS use one of ssl, tls, mqtts, mqtt+ssl or tcps.
Example: ✅
mqtts://test.mosquitto.org
In my case, enable the IIS server & then restart and check again.
We are using a SpringBoot service. Our restTemplate code looks like below:
#Bean
public RestTemplate restTemplate(final RestTemplateBuilder builder) {
return builder.requestFactory(() -> {
final ConnectionPool okHttpConnectionPool =
new ConnectionPool(50, 30, TimeUnit.SECONDS);
final OkHttpClient okHttpClient =
new OkHttpClient.Builder().connectionPool(okHttpConnectionPool)
// .connectTimeout(30, TimeUnit.SECONDS)
.retryOnConnectionFailure(false).build();
return new OkHttp3ClientHttpRequestFactory(okHttpClient);
}).build();
}
All our call were failing after the ReadTimeout set for the restTemplate. We increased the time, and our issue was resolved.
This error occurred in my application with the CIP-protocol whenever I didn't Send or received data in less than 10s.
This was caused by the use of the forward open method. You can avoid this by working with an other method, or to install an update rate of less the 10s that maintain your forward-open-connection.

Categories