We Keep Coding

C# HttpClient not using all established connections to a host with parallel requests

I'm using .NET Framework 4.8 for a console application that manages an ETL process and a .NET standard 2.0 library for the HTTP requests that use the HttpClient. This application is expected to handle millions of records and is long running.
These requests are made in parallel with a maximum concurrency limit of 20.
At application launch, I increase the number of connections .NET can make to a single host in a connection pool via the ServicePointManager. This is a frequent cause for connection pool starvation, as in .NET Framework it defaults to 2.
public static async Task Main(string[] args)
{
ServicePointManager.DefaultConnectionLimit = 50;
...
}
This is greater than the number of max concurrent requests allowed to ensure my requests are not being queued up on connections that are already being used.
I then loop through the records and post them to a TPL Dataflow block with the concurrency limit of 20.
This block makes the request on my API client which uses a singleton HttpClient for all requests..
public class MyApiClient
{
private static HttpClient _httpClient { get; set; }
public MyApiClient()
{
httpClient = new HttpClient();
}
public async Task<ReturnedObject> UploadNewDocumentAsync(DocParams docParams, DocData docData)
{
MultipartFormDataContent content = ConstructMultipartFormDataContent(docParams);
HttpContent httpContent = docData.ConvertToHttpContent();
content.Add(httpContent);
using (HttpResponseMessage response = await _httpClient.PostAsync("document/upload", content))
{
return await HttpResponseReader.ReadResponse<ReturnedObject>(response).ConfigureAwait(false);
}
}
}
I'm using sysinternals TCPView.exe to view all connections made between the local host and remote host, as well as what the status of those connections are and whether data is actively being sent or not.
When starting the application I see new connections established for each request made, until around 50 connections are established. I also see activity for around 20 at any given time. This meets expectations.
After around 24 hours of activity, TCPView shows only 5 connections concurrently sending and receiving data. All 50 connections still exist and are all in the Established state, but the majority sit idle. I don't have a way of logging when connections stop being actively used. So I don't know whether all of a sudden it drops from using 20 connections to only 5, or whether it gradually decreases.
My log file records elapsed time for every request made and I also see a degradation in performance at this time. Requests take longer and longer to complete, and I see an increase in TaskCanceled exceptions as the HttpClient reaches its timeout value of 100 seconds.
I've also confirmed with the 3rd party API vendor that they are not receiving a large number of incoming requests timing out.
This suggests to me that the application is still trying to make 20 requests at a time, but they are being queued up on a smaller number of TCP connections. All the symptoms point to classic connection pool starvation.
While the application is running, I can output some information from the ServicePointManager to the console.
ServicePoint sp = ServicePointManager.FindServicePoint(myApiService.BasePath, WebRequest.DefaultWebProxy);
Console.WriteLine(
$"CurrentConnections: {sp.CurrentConnections} " +
$"ConnectionLimit: {sp.ConnectionLimit}");
Console output:
CurrentConnections: 50 ConnectionLimit: 50
This validates what I see in TCPView.exe, which is that all 50 connections are still allowed and are established.
The symptoms show connection pool starvation, but TCPView.exe and ServicePointManager show there are plenty of available established connections. .NET is just not using them all. This behaviour only shows up after several hours of runtime. The issue is repeatable. If I close and relaunch the application, it begins by rapidly opening all 50 TCP connections, and I see data being transferred on up to 20 at a time. When I check 24 hours later, the symptoms of connection pool starvation have shown up again.
What could cause the behavior, and is there anything further I could do to validate my assumptions?

gRPC connection gets cut after 60 seconds if inactivity

I have been trying to set up a gRPC API capable of streaming events to a client. Basically, after a client has subscribed, the server will use gRPC's "Server Streaming" feature to send any new event to the client.
I expect there to be periods of inactivity, where the connection should remain active. However, with my current setup it seems Nginx is cutting the connection after 60 seconds of inactivity with the following exception at the client:
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: The response ended prematurely, with at least 9 additional bytes expected.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
The question is why? and how can I prevent it?
My setup
The API is built in ASP.NET Core 3 (will probably upgrade to .NET 5 soon) and is running in a Docker container on a Digital Ocean server.
Nginx is also running in a Docker container on the server and works as a reverse proxy for the API (among other things).
The client is a simple C# client written in .NET Core and is run locally.
What have I tried?
I have tried to connect to the Docker image directly on the server using grpc_cli (bypassing Nginx) where the connection remain active for long periods of inactivity without any issues. So I can't see what else it can be, except Nginx. Also, most of Nginx default timeout values seem to be 60 seconds.
I have tried these Nginx settings and various combinations of them, yet haven't found the right one (or the right combination) yet:
location /commands.CommandService/ {
grpc_pass grpc://commandApi;
grpc_socket_keepalive on;
grpc_read_timeout 3000s; # These are recommended everywhere, but I haven't had any success
grpc_send_timeout 3000s; #
grpc_next_upstream_timeout 0;
proxy_request_buffering off;
proxy_buffering off;
proxy_connect_timeout 3000s;
proxy_send_timeout 3000s;
proxy_read_timeout 3000s;
proxy_socket_keepalive on;
keepalive_timeout 90s;
send_timeout 90s;
client_body_timeout 3000s;
}
The most common suggestion for people with similar issues is to use grpc_read_timeout and grpc_send_timeout, but they don't work for me. I guess it makes sense since I'm not actively sending/receiving anything.
My client code looks like this:
var httpClientHandler = new HttpClientHandler();
var channel = GrpcChannel.ForAddress("https://myapi.com", new GrpcChannelOptions()
{
HttpClient = new HttpClient(httpClientHandler) { Timeout = Timeout.InfiniteTimeSpan },
});
var commandService = channel.CreateGrpcService<ICommandService>();
var request = new CommandSubscriptionRequest()
{
HandlerId = _handlerId
};
var sd = new CancellationTokenSource();
var r = new CallContext(callOptions: new CallOptions(deadline: null, cancellationToken: sd.Token));
await foreach (var command in commandService.SubscribeCommandsAsync(request, r))
{
Console.WriteLine("Processing command: " + command.Id);
}
return channel;
To be clear, the call to the API works and I can receive commands from the server. If I just keep sending commands from the API, everything is working beautifully. But as soon as I stop for 60 seconds (I have timed it), the connection breaks.
A possible workaround would be to just keep sending a kind of heartbeat to keep the connection open, but I would prefer not to.
Does anyone know how I can fix it? Am I missing something obvious?
UPDATE: Turns out it wasn't Nginx. After I updated the API and the client to .NET 5 the problem disappeared. I can't say in what version this was fixed, but at least it's gone in .NET 5.

Not sure this is an Nginx issue, looks like a client connection problem.
Your results look very similar to an issue I had, that should have been fixed in .net 3.0 patch. Try updating to a newer version of .NET and see if that fixes the problem.
Alternatively, it could be a problem with the max number of connections. Try setting the MaxConcurrentConnections for the kestrel server (in appsettings.json):
{
"Kestrel": {
"Limits": {
"MaxConcurrentConnections": 100,
"MaxConcurrentUpgradedConnections": 100
}
}
}

IBM MQ XMS.NET client reconnect timeout can't be changed

IBM MQ has an automatic client renonnect functionality, and it has a default timeout of 30 minutes. After 30 minutes it stops trying to reconnect (source - p35).
I want to change the timeout so it lasts a longer time retrying (for example 2 hours). I suppose I can use the property XMSC.WMQ_CLIENT_RECONNECT_TIMEOUT for this because it's available in the XMSC class.
Testing
I can simulate a connection failure by blocking the port 1414 where the client application makes connection to IBM MQ. And for testing purposes, I lower the timeout value to 5 minutes (instead of 30 minutes).
What I can see in the logging is that the client application receives XMSException with reason code 2544 (reconnecting):
IBM.XMS.XMSException: MQ delivered an asynchronous event with completion code 1, and reason 2544.
XMSWMQ2014.explanation
XMSWMQ2014.useraction
Linked Exception : CompCode: 1, Reason: 2544
This occurs for 30 minutes, and after that, I get an XMSException with reason code 2009 (connection broken). And auto reconnect fails.
XMSException occurred: IBM.XMS.XMSException: MQ delivered an asynchronous event with completion code 2, and reason 2009.
XMSWMQ2014.explanation
XMSWMQ2014.useraction
Linked Exception : CompCode: 2, Reason: 2009
I can conclude that changing the timeout value has no effect... Am I configuring the reconnect timeout in a wrong way?
Below, there is a code snippet:
XMSFactoryFactory factory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
IConnectionFactory connectionFactory = factory.CreateConnectionFactory();
connectionFactory.SetStringProperty(XMSC.WMQ_HOST_NAME, "hostname");
connectionFactory.SetIntProperty(XMSC.WMQ_PORT, 1414);
connectionFactory.SetStringProperty(XMSC.WMQ_CHANNEL, "channel_name");
connectionFactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT_UNMANAGED);
connectionFactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, "*");
connectionFactory.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_OPTIONS, XMSC.WMQ_CLIENT_RECONNECT_Q_MGR);
connectionFactory.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_TIMEOUT, 300); //300 seconds = 5 minutes
IConnection conn = connectionFactory.CreateConnection();
conn.Start();
IBM MQ Client version: 8.0.0.5
Notes
If I unblock the port in time, it can successfully reconnect.
Official IBM documentation: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.msc.doc/xms_automatic_client_reconnection.htm

I've found a way to accomplish this, but unfortunately not by code...
The reconnect timeout can be set in mqclient.ini.
Example:
CHANNELS:
MQReconnectTimeout = 14400
With this configuration applied, the client application should keep retrying for 4 hours.

.NET WebSockets forcibly closed despite keep-alive and activity on the connection

We have written a simple WebSocket client using System.Net.WebSockets. The KeepAliveInterval on the ClientWebSocket is set to 30 seconds.
The connection is opened successfully and traffic flows as expected in both directions, or if the connection is idle, the client sends Pong requests every 30 seconds to the server (visible in Wireshark).
But after 100 seconds the connection is abruptly terminated due to the TCP socket being closed at the client end (watching in Wireshark we see the client send a FIN). The server responds with a 1001 Going Away before closing the socket.
After a lot of digging we have tracked down the cause and found a rather heavy-handed workaround. Despite a lot of Google and Stack Overflow searching we have only seen a couple of other examples of people posting about the problem and nobody with an answer, so I'm posting this to save others the pain and in the hope that someone may be able to suggest a better workaround.
The source of the 100 second timeout is that the WebSocket uses a System.Net.ServicePoint, which has a MaxIdleTime property to allow idle sockets to be closed. On opening the WebSocket if there is an existing ServicePoint for the Uri it will use that, with whatever the MaxIdleTime property was set to on creation. If not, a new ServicePoint instance will be created, with MaxIdleTime set from the current value of the System.Net.ServicePointManager MaxServicePointIdleTime property (which defaults to 100,000 milliseconds).
The issue is that neither WebSocket traffic nor WebSocket keep-alives (Ping/Pong) appear to register as traffic as far as the ServicePoint idle timer is concerned. So exactly 100 seconds after opening the WebSocket it just gets torn down, despite traffic or keep-alives.
Our hunch is that this may be because the WebSocket starts life as an HTTP request which is then upgraded to a websocket. It appears that the idle timer is only looking for HTTP traffic. If that is indeed what is happening that seems like a major bug in the System.Net.WebSockets implementation.
The workaround we are using is to set the MaxIdleTime on the ServicePoint to int.MaxValue. This allows the WebSocket to stay open indefinitely. But the downside is that this value applies to any other connections for that ServicePoint. In our context (which is a Load test using Visual Studio Web and Load testing) we have other (HTTP) connections open for the same ServicePoint, and in fact there is already an active ServicePoint instance by the time that we open our WebSocket. This means that after we update the MaxIdleTime, all HTTP connections for the Load test will have no idle timeout. This doesn't feel quite comfortable, although in practice the web server should be closing idle connections anyway.
We also briefly explore whether we could create a new ServicePoint instance reserved just for our WebSocket connection, but couldn't see a clean way of doing that.
One other little twist which made this harder to track down is that although the System.Net.ServicePointManager MaxServicePointIdleTime property defaults to 100 seconds, Visual Studio is overriding this value and setting it to 120 seconds - which made it harder to search for.

I ran into this issue this week. Your workaround got me pointed in the right direction, but I believe I've narrowed down the root cause.
If a "Content-Length: 0" header is included in the "101 Switching Protocols" response from a WebSocket server, WebSocketClient gets confused and schedules the connection for cleanup in 100 seconds.
Here's the offending code from the .Net Reference Source:
//if the returned contentlength is zero, preemptively invoke calldone on the stream.
//this will wake up any pending reads.
if (m_ContentLength == 0 && m_ConnectStream is ConnectStream) {
((ConnectStream)m_ConnectStream).CallDone();
}
According to RFC 7230 Section 3.3.2, Content-Length is prohibited in 1xx (Informational) messages, but I've found it mistakenly included in some server implementations.
For additional details, including some sample code for diagnosing ServicePoint issues, see this thread: https://github.com/ably/ably-dotnet/issues/107

I set the KeepAliveInterval for the socket to 0 like this:
theSocket.Options.KeepAliveInterval = TimeSpan.Zero;
That eliminated the problem of the websocket shutting down when the timeout was reached. But then again, it also probably turns off the send of ping messages altogether.

I studied this issue these days, compared capture packages in Wireshark(webclient-client of python and WebSocketClient of .Net), and found what happened. In WebSocketClient, "Options.KeepAliveInterval" only send one packet to the server when no message received from server in these period. But some server only judge if there is active message from client. So we have to manually send arbitrary packets (not necessarily ping packets,and WebSocketMessageType has no ping type) to the server at regular intervals,even if the server side continuously sends packets. That's the solution.

C# Web Reference Call Timeout

I am making a call to an external service using a web reference. The IP's are dynamic so I call them one by one, and everything works fine. Periodically some of the IP's won't be available and I am getting a timeout which I am handling. The issue is the length of time it takes to timeout is around 30 seconds for each call. I tried changing the timeout property on the ws to 5 seconds but it doesn't seem to make a difference. Can someone please help me with this?

You could perform the DNS lookup yourself with a shorter timeout (e.g. 1000 ms):
http://www.chapleau.info/cs/blogs/fchapleau/archive/2008/09/09/reverse-dns-lookup-with-timeout-in-c.aspx
And then (if a IP address was found) perform the Web Service call using the IP address (to avoid the DNS lookup where You cannot control the timeout) using a TimeOut of e.g. 4000 ms (or even better : 5000ms - (the time the DNS lookup took)) to achieve a total timeout of 5000 ms.

Here is the answer I was looking for:
Adjusting HttpWebRequest Connection Timeout in C#
****Important Snippet:****
From the MSDN documentation of the HttpWebRequest.Timeout property:
A Domain Name System (DNS) query may take up to 15 seconds to return or time
out. If your request contains a host name that requires resolution and you set Timeout to a value less than 15 seconds, it may take 15 seconds or more before a WebException is thrown to indicate a timeout on your request.

You say that you changed "the timeout property on the ws to 5 seconds". If you're changing the timeout on the server, then that's not going to help you when your client can't connect.
You're using a "web reference", so I assume you have a client class derived from System.Web.Services.Protocols.SoapHttpClientProtocol. That class derives from System.Web.Services.Protocols.WebClientProtocol, which has a Timeout property. Try changing that property on your client before making the call and see if you get better results.

It might help if you ping the IP address before creating a proxy object and calling the web service here is the details on Ping class http://msdn.microsoft.com/en-us/library/system.net.networkinformation.ping.aspx