We Keep Coding

Async-Await api performance bottleneck at DB in .net5.0 web api

.net5.0 Web API
As there were few API's which were extensively being used by the clients, all over performance was being deteriorated. So we decided to convert those selected API's to Async/Await to reduce the load at IIS.
After implementing the same, we got better performances at about 100 parallel requests (via jMeter) in our local environment. But as soon as we increased the load to 200 requests, it started giving the below error: "The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached."
We realized that we didn't exactly improve the performance but we shifted the bottleneck from IIS to DB.
To solve this we tried changing the connection string (My SQL Server) properties i.e Max Pool size, ConnectRetryCount, ConnectRetryInterval etc. which definitely worked and gave us better results but everything came with a trade-off. Increasing Max Pool size would utilize DB server resources.
Also, we can never predict how many parallel requests would come to our API i.e if my max pool size is 400, what if 450 parallel requests arrives, it would still break.
We identified few options like using SemaphoreSlime, to limit the number of requests reaching the DB which would not open too many connections and hence restricting the timeout errors. But here we are not able to utilize the async nature of my api to its fullest.
Is there a optimized solution to this, or are we missing something?
Also, how safe is SemaphoreSlime if we choose to use it?

IIS and Kestrel can be configured to limit the maximum number of connections, so why to use your own home made solution?
You would can achieve same goal by increasing the connection timeout instead of using SemaphoreSlim.
If you want to increase the throughput capacity of you app you should start optimizing your queries, if this is not enough you should consider increasing the database server hardware resources.
If you have 400 max pool size and 450 concurrent requests arrive, it would not necessarily break. If there is no available connections in the pool, Connection.OpenAsync will wait until a connection is available. So if the queries are fast enough, it will not time out.

Here are two suggestions that could improve the performance of the system:
Make the work shorter so that there is less parallel work:
Use a stored procedure that does the work locally rather that moving data between the api and the db
Scale the database up,
Optimize the queries
Confirm indexes
Use caching
etc.
Accept the requests but put them on a queue and make the consumer of the queue process them in batches rather than one by one.

What is the highest safe number for ServicePointManager.DefaultConnectionLimit in a .net core app?

I have a .net core api that must make around 150,000 calls to collect data from external services. I am running these requests in parallel using Parallel.forEach and that seems to be working great, however I get an error from the http client for around 100,000 of my requests!
The Operation was canceled
Looking back at this I wish I had also logged the exception type but I believe this is due to not having enough outgoing connection limit.
Through debugging I have found that this returns 2:
ServicePointManager.DefaultConnectionLimit
On the face of it, if this really is the maximum amount of open connections allowed to an external domain / server, I want to increase that as high as possible. Ideally to 150,000 to ensure parallel processing doesnt cause an issue.
The problem is I cant find any information on what a safe limit is, or how much load this will put on my machine - if it is even a lot. Since this issue causes a real request to be made my data provider counts it in my charges - but obviously I get nothing from it since the .net core framework is just throwing my result away..
Since this problem is also intermittent it can be difficult to debug and I would just like to set this value as high as is safe to do so on my local machine.
I believe this question is relevant to stackoverflow since it does deal directly with the technical issue above, whereas other questions I could find only ask details about what this setting is.

As far as I understand, you are trying to make 150000 simulatenous request to external services. I presume that your services are Restful web services. If that is the case when you set DefaultConnectionLimit to an arbitrary number (very high), every single request opens a port for requesting data. This definitely clogs your network and your ports (port range is 0 to 65535).
Besides, making 150000 request without using throttling uncontrollably consumes your OS resources.
DefaultConnectionLimit is there because it protects you from aforementioned problems.
you may consider to use SemaphoreSlim for throttling

HttpClient shared instance with different sessions

Context
The recommended way to use HttpClient is to create one instance and share it (according to the Microsoft documentation). There are many examples out there where using an HttpClient per request (on a server) yielded issues.
Problem
In my situation I have to run up to 20 simultaneous "sessions" which cannot interfere with one another. By "session" I mean isolated set of operations on a service or multiple services. No data should be shared between these sessions and specifically cookies.
These sessions are long lasting sessions (can last for days). But there may be only a maximum of 20 concurrent sessions at the same time. A slow start while instantiating these sessions is acceptable (up to 5 seconds).
Question
Should I use a pool? Should I reuse the same HttpClient instance? Should I spawn up to 20 HttpClients? Moreover, considering that they run concurrently, is it correct to assume that concurrent calls to a single HttpClient will be blocking?

The recommendation for creating only a single instance of HttpClient is because it creates a TCP connection that persists for a period of time. Eventually, it will get garbarge collected or closed naturally, but in the meantime, you could potentially exhaust the maximum amount of open connections on the server. There's also a bit of resource utilization from each connection that could prove problematic when stacked.
However, this is one of those things where there's no hard and fast rule. Certain things, like creating a new HttpClient per operation, are obviously problematic, but generally one instance per request to your server (not per request sent through HttpClient) shouldn't be problematic unless you're servicing a high number of concurrent requests.
You can also just make it a singleton across your application. HttpClient is not technically a user or session-specific thing. It would only be problematic if your changing default headers that should not be leaked across sessions. For example, you'd want to attach something like an Authorization header as part of creating a request, not as a default header when setting up the client. As long as you're smart about how you use it, there's no problem having it shared. That being "smart about how you use it" thing can sometimes be a point of failure, though. Even if you are, will the next developer be. More to the point, will they even know they should be? You might even forget, yourself.
Personally, I'd start making it request-scoped, where each request to your web application gets a separate instance, but that same instance will be used for all operations with HttpClient during that request. You can then do some profiling to see how the server is handling this, and if it does become a problem, then you can look for alternative solutions.

What are the downsides to Request throttling using delay (C# .Net 4 Web Server)

We are running a Http Api and want to be able to set a limit to the number of requests a user can do per time unit. When this limit has been reached, we don't want the users to receive errors, such as Http 429. Instead we want to increase the response times. This has the result that the users can continue to work, but slower, and can then choose to upgrade or not upgrade its paying plan. This solution can quite easily be implemented using Thread.sleep (or something similar) for x number of seconds, on all requests of a user that has passed its limit.
We think that in worst case there might be a problem with the number of possible connections for a single server, since as long as we keep delaying the response, we keep a connection open, and therefore limiting the number of possible other connections.
All requests to the Api is running asynchronously. The Server itself is built to be scalable and is running behind a load balancer. We can start up additional servers if necessary.
When searching for this type of throttling, we find very few examples of this way of limiting the users, and the examples we found seemed not concerned at all about connections running out. So we wonder is this not a problem?
Are there any downsides to this that we are missing, or is this a feasible solution? How many connections can we have open simultaneously without starting to get problems? Can our vision be solved in another way, that is without giving errors to the user?

Thread.Sleep() is pretty much the worst possible thing you can do on a web server. It doesn't matter that you are running things asynchronously because that only applies to I/O bound operations and then frees the thread to do more work.
By using a Sleep() command, you will effectively be taking that thread out of commission for the time it sleeps.
ASP.Net App Pools have a limited number of threads available to them, and therefore in the worst case scenario, you will max out the total number of connections to your server at 40-50 (whatever the default is), if all of them are sleeping at once.
Secondly
This opens up a major attack vector in terms of DOS. If I am an attacker, I could easily take out your entire server by spinning up 100 or 1000 connections, all using the same API key. Using this approach, the server will dutifully start putting all the threads to sleep and then it's game over.
UPDATE
So you could use Task.Delay() in order to insert an arbitrary amount of latency in the response. Under the hood it uses a Timer which is much lighter weight than using a thread.
await Task.Delay(numberOfMilliseconds);
However...
This only takes care of one side of the equation. You still have an open connection to your server for the duration of the delay. Because this is a limited resource it still leaves you vulnerable to a DOS attack that wouldn't have normally existed.
This may be an acceptable risk for you, but you should at least be aware of the possibility.

Why not simply add a "Please Wait..." on the client to artificially look like it's processing? Adding artificial delays on server costs you, it leaves connections as well as threads tied up unnecessarily.

Tips / techniques for high-performance C# server sockets

I have a .NET 2.0 server that seems to be running into scaling problems, probably due to poor design of the socket-handling code, and I am looking for guidance on how I might redesign it to improve performance.
Usage scenario: 50 - 150 clients, high rate (up to 100s / second) of small messages (10s of bytes each) to / from each client. Client connections are long-lived - typically hours. (The server is part of a trading system. The client messages are aggregated into groups to send to an exchange over a smaller number of 'outbound' socket connections, and acknowledgment messages are sent back to the clients as each group is processed by the exchange.) OS is Windows Server 2003, hardware is 2 x 4-core X5355.
Current client socket design: A TcpListener spawns a thread to read each client socket as clients connect. The threads block on Socket.Receive, parsing incoming messages and inserting them into a set of queues for processing by the core server logic. Acknowledgment messages are sent back out over the client sockets using async Socket.BeginSend calls from the threads that talk to the exchange side.
Observed problems: As the client count has grown (now 60-70), we have started to see intermittent delays of up to 100s of milliseconds while sending and receiving data to/from the clients. (We log timestamps for each acknowledgment message, and we can see occasional long gaps in the timestamp sequence for bunches of acks from the same group that normally go out in a few ms total.)
Overall system CPU usage is low (< 10%), there is plenty of free RAM, and the core logic and the outbound (exchange-facing) side are performing fine, so the problem seems to be isolated to the client-facing socket code. There is ample network bandwidth between the server and clients (gigabit LAN), and we have ruled out network or hardware-layer problems.
Any suggestions or pointers to useful resources would be greatly appreciated. If anyone has any diagnostic or debugging tips for figuring out exactly what is going wrong, those would be great as well.
Note: I have the MSDN Magazine article Winsock: Get Closer to the Wire with High-Performance Sockets in .NET, and I have glanced at the Kodart "XF.Server" component - it looks sketchy at best.

Socket I/O performance has improved in .NET 3.5 environment. You can use ReceiveAsync/SendAsync instead of BeginReceive/BeginSend for better performance. Chech this out:
http://msdn.microsoft.com/en-us/library/bb968780.aspx

A lot of this has to do with many threads running on your system and the kernel giving each of them a time slice. The design is simple, but does not scale well.
You probably should look at using Socket.BeginReceive which will execute on the .net thread pools (you can specify somehow the number of threads it uses), and then pushing onto a queue from the asynchronous callback ( which can be running in any of the .NET threads ). This should give you much higher performance.

A thread per client seems massively overkill, especially given the low overall CPU usage here. Normally you would want a small pool of threads to service all clients, using BeginReceive to wait for work async - then simply despatch the processing to one of the workers (perhaps simply by adding the work to a synchronized queue upon which all the workers are waiting).

I am not a C# guy by any stretch, but for high-performance socket servers the most scalable solution is to use I/O Completion Ports with a number of active threads appropriate for the CPU(s) the process s running on, rather than using the one-thread-per-connection model.
In your case, with an 8-core machine you would want 16 total threads with 8 running concurrently. (The other 8 are basically held in reserve.)

The Socket.BeginConnect and Socket.BeginAccept are definitely useful. I believe they use the ConnectEx and AcceptEx calls in their implementation. These calls wrap the initial connection negotiation and data transfer into one user/kernel transition. Since the initial send/recieve buffer is already ready the kernel can just send it off - either to the remote host or to userspace.
They also have a queue of listeners/connectors ready which probably gives a bit of boost by avoiding the latency involved with userspace accepting/receiving a connection and handing it off (and all the user/kernel switching).
To use BeginConnect with a buffer it appears that you have to write the initial data to the socket before connecting.

As others have suggested, the best way to implement this would be to make the client facing code all asynchronous. Use BeginAccept() on the TcpServer() so that you dont have to manually spawn a thread. Then use BeginRead()/BeginWrite() on the underlying network stream that you get from the accepted TcpClient.
However, there is one thing I dont understand here. You said that these are long lived connections, and a large number of clients. Assuming that the system has reached steady state, where you have your max clients (say 70) connected. You have 70 threads listening for the client packets. Then, the system should still be responsive. Unless your application has memory/handle leaks and you are running out of resources so that your server is paging. I would put a timer around the call to Accept() where you kick off a client thread and see how much time that takes. Also, I would start taskmanager and PerfMon, and monitor "Non Paged Pool", "Virtual Memory", "Handle Count" for the app and see whether the app is in a resource crunch.
While it is true that going Async is the right way to go, I am not convinced if it will really solve the underlying problem. I would monitor the app as I suggested and make sure there are no intrinsic problems of leaking memory and handles. In this regard, "BigBlackMan" above was right - you need more instrumentation to proceed. Dont know why he was downvoted.

Random intermittent ~250msec delays might be due to the Nagle algorithm used by TCP. Try disabling that and see what happens.

One thing I would want to eliminate is that it isn't something as simple as the garbage collector running. If all your messages are on the heap, you are generating 10000 objects a second.
Take a read of Garbage Collection every 100 seconds
The only solution is to keep your messages off the heap.

I had the same issue 7 or 8 years ago and 100ms to 1 sec pauses , the problem was Garbage Collection .. Had about 400 Meg in use from 4 gig BUT there were a lot of objects.
I ended up storing messages in C++ but you could use ASP.NET cache ( which used to use COM and moved them out of the heap )

I don't have an answer but to get more information I'd suggest sprinkling your code with timers and logging avg and max time taken for suspect operations like adding to the queue or opening a socket.
At least that way you will have an idea of what to look at and where to begin.