We Keep Coding

How does ServicePointManager.ReusePort and SO_REUSE_UNICASTPORT alleviate ephemeral port exhaustion?

Windows 10 and Windows Server 2016 introduced the SO_REUSE_UNICASTPORT socket option. It was made available for use in .NET starting from version 4.6 via the ServicePointManager.ReusePort static property. I'm suffering from ephemeral port exhaustion on a .NET application during very high loads (many concurrent outgoing requests via HttpClient), and I am considering using this option to deal with it. I'm aware of other ways to deal with the problem (such as editing the Windows Registry to modify the max number of ephemeral ports or shortening TIME_WAIT), but I'd like to fully under this solution too.
The documentation for ServicePointManager.ReusePort is very minimal:
Setting this property value to true causes all outbound TCP connections from HttpWebRequest to use the native socket option SO_REUSE_UNICASTPORT on the socket. This causes the underlying outgoing ports to be shared. This is useful for scenarios where a large number of outgoing connections are made in a short time, and the app risks running out of ports.
Looking at the documentation for SO_REUSE_UNICASTPORT doesn't provide any additional insights:
When set, allow ephemeral port reuse for Winsock API connection functions which require an explicit bind, such as ConnectEx. Note that connection functions with an implicit bind (such as connect without an explicit bind) have this option set by default. Use this option instead of SO_PORT_SCALABILITY on platforms where both are available.
I couldn't find any explanation on the web as to how exactly this "ephemeral port reuse" is achieved, how exactly it works on a technical level, and how well it reduces the risk of ephemeral port exhaustion. How much of an improvement can I expect? Using this feature, how can I calculate a new limit for my application? Are there any downsides to enabling this?
This is all shrouded in mystery and I'd love it if someone can explain this new mechanism and its implications.

TCP connection is uniquely identified by (local IP, local port, remote IP, remote port). That means it's perfectly possible to use the same (local IP, local port) pair for multiple sockets connecting to different remote endpoints. Suppose you want to make http request to "site1.com" and "site2.com". You are using sockets with the following code:
using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) {
socket.Bind(new IPEndPoint(IPAddress.Parse("some local ip"), 45455));
socket.Connect(server, port);
socket.Send(someBytes);
// ...
}
So you are binding socket to specific local endpoint with port 45455. If you now try to do that concurrently to make requests to "site1.com" and "site2.com" you will get "an address already in use" exception.
But if you add ReuseAddress option (note that it's not option your question is about) before binding:
socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
You will be able to bind to sockets to the same local (ip, port) and you will see in netstat two ESTABLISHED connections.
All of the above is to show that in theory nothing prevents one to reuse even ephemeral port to make multiple connections to different remote endpoints. But, when you bind to ephemeral port (0) - it is not yet known to which remote endpoint you are going to connect. Suppose all ephemeral ports are in use and you are binding to 0. OS provides you some port for reuse during bind phase, there is one socket using this port already connected to "site1.com". You are trying to connect to "site1.com" too and that fails (because all 4 values identifying tcp connection are the same for both sockets).
What SO_REUSE_UNICASTPORT does is it delays choosing ephemeral port when you bind to 0 until actual connection phase (Connect() call for example). At this phase (unlike bind) you already know local ip, remote ip, remote port you are going to connect to, and you need to choose ephemeral port. Suppose all ports are in use. Now, instead of choosing some random port for reuse (and potentially fail later on connect) you can choose port which is connected to different remote endpoint (different from what current socket is trying to connect to).
You can confirm this for example at this MS support article:
SO_REUSE_UNICASTPORT
For a connection scenario to be implemented, the socket option must be
set before binding a socket. This option instructs the system to
postpone port allocation until connection time when the 4-tuple
(quadruple) for the connection is known.
Note that SO_REUSE_UNICASTPORT only has effect on explicit bindings (as stated in your question quote, but still worth repeating). If you bind implicitly (such as when you just Connect() without binding) - this option is already set by default (where supported of course).
About which effect this has on your particular application. First from the above it should be clear that if your application makes ton of requests to the same remote endpoint (to the same http server for example) - this option will have no effect. If you make a lot of requests to different endpoints though - it should help to prevent ports exhaustion. Still effect of ServicePointManager.ReusePort itself depends on how HttpClient works with sockets internally I guess. If it just Connect() without explicit binding - this option should be enabled (on supported systems) by default, so setting ServicePointManager.ReusePort to true will not have additional effect, otherwise it will. Since you don't know (and should not rely on) its internal implementation its worth enabling ServicePointManager.ReusePort in your particular scenario.
You can also perform tests with this option on\off by limiting range of ephemeral ports (with command like netsh int ipv4 set dynamicport tcp) to some small amounts and see how it goes.

SQL refusing connection in load test

I'm running a load test on my system. At a certain level of load, I start getting SQL errors in my log:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Prprovidererror: 40 - Could not operrorconnection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The network path was not found
By running performance monitor on the SQL server in question, I found the following:
CPU level rarely exceeds 50%. (On a previous iteration I saw that it was maxing out at 100%, so I increased the specs of the VM, which helped push the problem to a higher load level.)
Number of user connections got to a shade over 8,000. The Sql Server has the default setting of 32,767 max connections.
The connection string specifies a max pool size of 1000 connections to each database, and there are 100 databases on the server. The load test is randomly distributed between the 100 databases, so there should be a fairly even distribution, meaning about 80 connections per database. Nowhere near the 1k limit.
What other factors might cause Sql Server to stop being able to accept connections?
UPDATE: extra info: I'm using Entity Framework Core (EF7) for my DB connections, if that helps any.

"Network Path Not Found" does not seem like an error related to SQL Server's capacity. As a former "IT Guy," I suspect that a firewall is dropping your packets. If this is during a stress test, the firewall could be interpreting the numerous requests as a denial of service attack, and using some sort of predefined rule to drop connections for a specified time period.
What is your network environment? If you have a hardware firewall or router with IPS capabilities, I would check those logs to see if you find a smoking gun. You might have to create a special rule to allow unlimited traffic to your SQL Server.

It's a bit curious that you are getting that many connections to the database. You should be utilizing connection pooling; even under intense load, the connection pooling should greatly reduce the number of active connections being used.
Can you provide the code that's accessing the database? Are you calling the dispose() method or closing the connection?
Also, have you looked to see if data datacaching would ease the db load? A 2-5 second datacache can greatly reduce database calls.

You are running into the TCP listen() backlog limit for the SQL-Server's listening port. When this happens, Windows platforms (but not *nix platforms) will issue 'connection refused' for further incoming connections.
I'm not an SQL-Server guy but there is bound to be a parameter somewhere by which you can increase its listen backlog.
Alternatively you should look into better or more connection pooling at the client.

It turns out the problem wasn't on SQL at all. The problem was on our API server, where some of the APIs were spinning off hundreds of parallel threads, each making its own connection to the database. The load was simply too much for the API server, and it started returning "Access Denied" exceptions without even really attempting to connect to the database.
Solution: we throttled the number of threads being spun off, using the pattern shown in this answer.

How do I specify a valid port when testing a client and server socket application?

I'm trying to connect a client to a server test program in C# using Sockets on my local machine. However, I get the "server actively refused the connection" error for every port number I try.
How do I pick a valid port number to be able to just test out sockets between a sample client and server? Is there a valid port no. I can use for this scenario?

The most likely reasons you might not be able to bind to your desired port on the server of your choice:
Your system has a firewall configured to prevent outbound connections on the port in question.
Your network imposes such a restriction, either via a firewall or proxy/gateway.
The server's network won't allow inbound connections on the port in question, either via firewall or proxy/gateway.
The server itself is firewalled and won't allow inbound connections on the port in question.
Finally, it's always possible that the daemon in question on the server isn't running.
Check your ports:
0–1023 (0x000–0x03FF)are reserved for "well-known ports" and typically have certain restrictions, at least for the server end of things. It wouldn't make much sense to require special permissions for a client to use a well-known port.
1024–65535 (0x0400–0xFFFF)are putatively open for anybody to use.
However...every system has a range of ports designated as ephemeral ports that should not normally be used. Berkeley sockets used 1024–4999 as the emphemeral range, but that range varies from system to system and is [usually] configurable.

Socket Connection Stability

I have an async socket Server written in C#. Often, my users complain that they are disconnected from the without any reason. Other internet applications on the user's computer (MSN, SKYPE Etc.) appear to be well connected and therefore it does not seem to be an issue with their internet connection. Can anyone provide some information on how to make a connection between client and server rock solid (i mean prevent frequent disconnection) or where I can look for disconnection issues?

Network Address Translators (aka "home routers", "wireless gateways", etc.) that share an ISP assigned IP address among multiple computers may be at play here. NATs keep a table of active connections (sometimes known as a "port mapping table") that keeps track of how to translate the ip:port pair for active TCP connections and observed UDP sessions. There are some brands of routers that are known to clear the port mapping out of memory if it doesn't observe any traffic going over the TCP connection for some period of time. Apps like MSN Messenger and Skype mitigate this by periodically sending "keep alive" between client, server, and other persistent connections. These messages go in both directions.
You didn't give much information to go on, so the NAT problem is just one possibility out of many.
The best way to diagnose connectivity failures is to invest heavily in logging network messages (both on client and server), socket events, and all return values from the socket APIs. That way, if a customer reports a problem, he can send you his logs - and hopefully you can fetch the corresponding server logs for comparison. You can then diagnose where the disconnect originated from and/or what socket error reset the connection. There's a high probability something in your code is triggering the disconnect.

I assume you are talking about TCP here. The socket library which the clients of your application use is pretty much the same as the one that they use to connect to Skype or MSN. Usually the reason for disconnect is that one of the parties closes the socket (e.g. after connection is idle for some time). Check the places in your code where you close your connections.

High performance C# TCP server problem: No connection could be made because the target machine actively refused it

I have developed a TCP server according to your advises: High performance TCP server in C#
It is based on asynchron pattern.
I also developed a stress test application to test its performance. My server can get thousands of connections paralelly from my stress test app, can parse data and save it to my database.
When I stress my server, I can get "System.Net.Sockets.SocketException "No connection could be made because the target machine actively refused it" error from my server, so I have to reconnect to it. If I test it with 5000 concurrent connections, I have to try connect again because of this problem 10-20% of the connections, if I test it with 10K concurrent connections, it can be 30-40%. Sometimes it can be - very rarely - more, than 50%. It seems it can not handle connection accepts: I make new connections from my stress test as heavily as my test machine can - about 120 connections/sec.
So, what can cause this kind of exception? How to handle it? What to do in server side implementation to avoid this problem? How to tune TCP connection accept?
Thanks in advance!

You might be running out of available ports every now and then. You can view this easily using SysInternals' TcpView utility.
On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.
There are two registry entries that can reduce this time interval:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay
and increase the max number of ports that can be opened by an app:
HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort
EDIT: MaxFreeTcbs seems to be a third setting which could help (I haven't tried this yet), mentioned in this TechNet article which has more advice on tracking down odd network problems. HTH.

You are making connections faster than the software can listen for new connections, or in other words you are reaching the connections per second limit of that port. I think you can double the amount of connections per second by listening to a second port, client side you should just reconnect when you get the exception.
There are also limits applied to the amount of connection, for these see Chris O's answer.