We Keep Coding

Increase the request time-out of a request in ASP.NET CORE REST API

I'm quite a beginner in using HTTP Request,
I have an automated task that updates the attendance of 6000 employees every hour. When I was testing my API in my local, it succeeded with minimum 40 seconds.
Once I have deployed it, it gives me an operation time-out.
I wish to increase my Request Time-out to at least a minute.
I have used this code :
[HttpPost("autoAttendanceUpdate")]
public bool AUTO_AttendanceUpdate([FromBody] AttendanceProcessorFilters filter)
{
HttpClient httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMinutes(1);
_attendanceProcessorRepository.AUTO_attendanceUpdate(filter); // AUTO_attendanceUpdate contains the logic
}
But I think I have used httpClient wrong.
I don't know if I should modify AUTO_attendanceUpdate or if there are other solutions that can help me.
AS much as possible I don't want any other methods to be affected so adjusting web config is not an option. TIA

.NET Core service stalling under load due to HttpWebRequest

I had a ASP.NET (.NET Framework 4.8) web service running on a Windows Server, that made lots of outgoing HTTP requests using HttpWebRequest (synchronous). It was handling thousands of concurrent requests without any trouble.
Recently, I migrated the service/middleware to ASP.NET Core (runtime 3.1) running on Ubuntu Server, using the updated HttpWebRequest (synchronous).
Now this service is stalling under a load test with just a few hundred concurrent requests. System journal/logs indicate that health check (heartbeat) cannot reach the service after a few minutes. It starts out fine, but after a few minutes it slows down and eventually halts (no response but it doesn't crash dotnet though), and then starts working again after 5-10 minutes without any intervention, and repeats this same behavior every few minutes.
I'm not sure if this is due to port exhaustion or a deadlock. If I load test the service by skipping all HttpWebRequest calls, then it works fine, so I'm suspecting it has to do something with HttpWebRequest causing an issue under stress of traffic.
Looking at the .NET Core codebase, it seems like HttpWebRequest (synchronous) creates a new HttpClient for each request (client is not cached due to parameters in my case), and executes HttpClient synchronously like:
public override WebResponse GetResponse()
{
:
:
return SendRequest(async: false).GetAwaiter().GetResult();
:
:
}
private async Task<WebResponse> SendRequest(bool async)
{
:
:
_sendRequestTask = async ?
client.SendAsync(...) :
Task.FromResult(client.Send(...));
HttpResponseMessage responseMessage = await _sendRequestTask.ConfigureAwait(false);
:
:
}
Official suggestion from Microsoft is to use IHttpClientFactory or SocketsHttpHandler for better performance. I can make our service use a singleton SocketsHttpHandler and new HttpClient's per outgoing request (with the shared handler) to reuse & close sockets better, but my main concern is (below):
The service is based on synchronous code, so I'll have to use asynchronous HttpClient synchronously, probably using the same method.GetAwaiter().GetResult() technique as official .NET Core code above. While a singleton SocketsHttpHandler may help avoid port exhaustion, could concurrent synchronous execution still result in the stalling problem due to deadlocks like the native HttpWebRequest?
Also, is there an approach (another synchronous HTTP client for .NET Core, setting 'Connection: close' header etc.) to smoothly making lots of concurrent HTTP requests synchronously without port exhaustion or deadlocks, just like it worked smoothly earlier with HttpWebRequest in .NET Framework 4.8?
Just to clarify, all WebRequest related objects are closed/disposed properly in the code, ServicePointManager.DefaultConnectionLimit is set to int.MaxValue, nginx (proxy to dotnet) has been tuned, sysctl has been tuned as well.

I'm not sure if this is due to port exhaustion or a deadlock.
Sounds more like thread pool exhaustion to me.
The service is based on synchronous code, so I'll have to use asynchronous HttpClient synchronously
Why?
The best solution to thread pool exhaustion is to rewrite blocking code to be asynchronous. There were places in ASP.NET pre-Core that required synchronous code (e.g., MVC action filters and child actions), but ASP.NET Core is fully asynchronous, including the middleware pipeline.
If you absolutely cannot make the code properly asynchronous for some reason, the only other workaround is to increase the minimum number of threads in the thread pool on startup.

Cannot read request body in Owin (stuck in ReadAsStringAsync)

Running OWIN self-hosting web service in Azure Service Fabric behind Azure Load Balancer and Azure Traffic Manager Profile with multiple instances.
About 2% of web requests stuck when service is trying to read the request body from incoming requests to string using HttpContent.ReadAsStringAsync. So the ReadAsStringAsync fails after timeout :
System.Net.Http.HttpRequestException: Error while copying content to a stream.
Web service receives request with different sizes most of them are small but some could be up to 60K. The number of concurrent pending requests/instance is about 900 and CPU load ~30%.
Scaling out number of services helps a little bit but does not solve it completely, I am also trying to understand what is the root cause.
Here is the setting I override in OwinHttpListener:
listener.SetRequestProcessingLimits(Environment.ProcessorCount * 5, int.MaxValue);
listener.SetRequestQueueLimit(10000);
Any ideas of the root cause and/or how to fix it?
Owin ver:
PackageReference Include="Microsoft.AspNet.WebApi.Owin" Version="5.2.6"

wcf service long httpwebrequest wait causes queuing of subsequent requests

I have a WCF service that in functionA makes an HttpWebRequest call to functionX in an external service. Originally the timeout on this httpwebrequest was set to 5 minutes.
Recently, the external service has been taking longer than 5 minutes to respond (which I am ok with). So I bumped the httpWebRequest.timeout up to 10 minutes.
Meanwhile the wcf service should be able to process other incoming requests (to functionB, functionC, etc). What I'm experiencing now is that if functionX takes longer than ~5 minutes to respond (and thus functionA takes longer than 5 minutes to complete), subsequent requests to functionB in my wcf service are queued / do not process until functionA completes.
In the end everything completes properly, but I don't see why functionB is affected by the waiting that is happening over in functionA.
Forgive me if that is hard to follow. It is a strange and I'm having trouble wrapping my head around how these pieces are related.

You must decorate your WCF Service class with following attribute
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)] // The service instance is multi-threaded.
public class Service1
{
// ...
}
I assume your concurrency mode is set to Single defined as follows by Microsoft.
"The service instance is single-threaded and does not accept reentrant calls.
If the System.ServiceModel.ServiceBehaviorAttribute.InstanceContextMode property is System.ServiceModel.InstanceContextMode.Single, and additional messages arrive while the instance services a call, these messages must wait until the service is available or until the messages time out."

i had a same problem. i hosted my service in IIS. after little search i found out its because of maxconnection limit in web config. i added this line in to my web.config and the problem solved:
<system.net>
<connectionManagement>
<add address="*" maxconnection="1000"/>
</connectionManagement>
</system.net>
by default maxconnection value is 2.
but this is one of the many reasons. you should monitor your server requests in order to find out the exact reason.

Periodic slowdown performance in ServiceStack

We have a web service using ServiceStack (v3.9.60) that is currently gets an average (per New Relic monitoring) of 600 requests per minute (load balanced with two Windows 2008 web servers.)
The actual time spend in the coded request Service (including Request Filter) takes about an average of 5ms (From what we see from recorded log4net logs.) It is offloading the request to an ActiveMQ endpoint and automatic have ServiceStack generate a 204 (Return204NoContentForEmptyResponse enabled with "public void Post(request)")
On top of that we have:
PreRequestFilters.Insert(0, (httpReq, httpRes) =>
{
httpReq.UseBufferedStream = true;
});
since we use the raw body to validate a salted hash value (passed as a custom header) during a Request Filter for approval reasons that it comes from a correct source.
Overall we see in New Relic that the whole web service call takes an average around 700ms, which is a lot compared to the 5ms it actually takes to perform the coded process. So when we looked deeper in the data New Relic reports we saw some requests periodically take quite some time (10-150 seconds per request.) Drilling down in the reporting of New Relic we see that Applying the Pre-Request Filter takes time (see image below.) We were wondering why this could be the case and if it was related to the buffered stream on the Http Request object and what possibly could be done to correct this?
EDIT
Have been playing around with this some and still haven't found an answer.
Things done:
Moved the Virtual Folder out from a sub-folder location of the actual site folder (there are about 11 other Web Services located under this site)
Assigned this Web Service to use its own Application Pool so it is not shared with the main site and other Web Services under the site
Added the requirement to Web.Config for usage of Server GC as Phil suggested
Disabled the pre-request filter that turned on the usage of buffered stream (and bypass the code that used the RawBody)
Added more instrumentation to New Relic for a better drill-down (see image below)
I'm starting to wonder if this is a Windows Server/IIS limitation due to load. But would like to hear from someone that is more familiar with such.