We Keep Coding

IIS worker thread vs web application thread

I'm maintaining an ASP.NET Core web application that needs to repeatedly run some background threads. I know it's not a good design but currently I have to fix its major issues with minimum effort. Now I wonder if I should worry about handling users http requests by web server or not?
Question is simple but I can't find any clear answer for it:
What is the difference between threads that are created in application like this:
Task.Run(() => { // some parallel job })
and worker threads of IIS that handle http requests?
Are they come from the same thread pool or they're reside in separate pools?

According to this it's all one pool: "ASP.NET Core already runs app code on normal Thread Pool threads." In other words, there isn't a separate max for threads serving requests vs. background threads.
The biggest difference is that IIS is aware of the threads it creates itself for an incoming request. IIS is not aware of any threads you create yourself.
When an app pool is recycled, or IIS is shut down, it waits until all requests have finished processing - it waits until the threads it creates for each request has finished processing - and then it kills the process. If you create any threads that outlive the request (for example, if you create a background thread and then send the response back to the client) IIS has no idea that thread is still running and could kill the whole process at any time.
If you don't return a response until all the threads are complete, then you won't have that specific problem.
The other issue is that you may hit the maximum number of allowable threads. Then all kinds of weird performance issues would happen. But that depends on how many threads you are creating and how many HTTP requests are coming in.

SignalR IIS worker process does not released

We have a simple signalr server and client running with the backplane enabled. When I looked in to the IIS worker process I found out in the current requests tab there is always this signalr connect is showing.
When I connect like 100 clients 100 current requests are shown in the woker process view. Shouldn't these be removed after connecting or is this the expected behavior from the signalr?

The threads will close or get reused by new clients, its basically a thread pool, you shouldn't need to worry about it. The only time you need worry about how many threads are open is if you only have limited cpu resources, and a low open thread limit set.

What could be rate limiting CPU cycles on my C# WCF Service?

Something very strange started happening on our production servers a day or two ago regarding a WCF Service we run there: it seems that something started rate limiting the process in question's CPU cycles to the amount of CPU cycles that would be available on one core, even though the load is spread across all cores (the process is not burning one core to 100% usage)
The Service is mostly just a CRUD (create, read, update, delete) service, with the exception of a few long running (can take up to 20 minutes) service calls that exist there. These long running service calls kicks of a simple Thread and returns void so not to make the Client application wait, or hold up the WCF connection:
// WCF Service Side
[OperationBehavior]
public void StartLongRunningProcess()
{
Thread workerThread = new Thread(DoWork);
workerThread.Start();
}
private void DoWork()
{
// Call SQL Stored proc
// Write the 100k+ records to new excel spreadsheet
// return (which kills off this thread)
}
Before the above call is kicked off, the service seems to respond as it should, Fetching data to display on the front-end quickly.
When you kick off the long running process, and the CPU usage goes to 100 / CPUCores, the front-end response gets slower and slower, and eventually wont accept any more WCF connections after a few minutes.
What I think is happening, is the long running process is using all the CPU cycles the OS is allowing, because something is rate limiting it, and WCF can't get a chance to accept the incoming connection, never mind execute the request.
At some point I started wondering if the Cluster our virtual servers run on is somehow doing this, but then we managed to reproduce this on our development machines with the client communicating to the service using the loopback address, so the hardware firewalls are not interfering with the network traffic either.
While testing this inside of VisualStudio, i managed to start 4 of these long running processes and with the debugger confirmed that all 4 are executing simultaneously, in different threads (by checking Thread.CurrentThread.ManagedThreadId), but still only using 100 / CPUCores worth of CPU cycles in total.
On the production server, it doesn't go over 25% CPU usage (4 cores), when we doubled the CPU cores to 8, it doesn't go over 12.5% CPU usage.
Our development machines have 8 cores, and also wont go over 12.5% CPU usage.
Other things worth mentioning about the service
Its a Windows Service
Its running inside of a TopShelf host
The problem didn't start after a deployment (of our service anyway)
Production server is running Windows Server 2008 R2 Datacenter
Dev Machines are running Windows 7 Enterprise
Things that we have checked, double checked, and tried:
Changing the process' priority up to High from Normal
Checked that the processor affinity for the process is not limiting to a specific core
The [ServiceBehavior] Attribute is set to ConcurrencyMode = ConcurrencyMode.Multiple
Incoming WCF Service calls are executing on different threads
Remove TopShelf from the equation hosting the WCF service in just a console application
Set the WCF Service throttling values: <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="1000" maxConcurrentSessions="1000" />
Any ideas on what could be causing this?

There must be a shared resource that only allows a single thread to access it at a time. This would effectively only allow one thread at a time to run, and create exactly the situation you have.
Processor affinity masks are the only way to limit a process to a single CPU, and if you did this you would see one CPU pinned and all the others idle (which is not your situation).
We use a tool called LeanSentry that is very good at identifying these kinds of problems. It will attach itself to IIS as a debugger and capture stack dumps of all executing processes, then tell you if most of your threads are blocked in the same spot. There is a free trial that would be long enough for you to figure this out.

The CPU usage looks like a lock on a table in the SQL Database to me. I would use the SQL management studio to analyze the statements see if it can confirm that.
Also you indicated that you call a stored procedure might want to have it look at that as well.
This all just looks like a database issue to me

More info from ThreadAbortException

We have an IIS-hosted WCF service that receives a large chunk of data to work on. The service fires up several worker threads and then returns leaving the worker threads to finish the job (which might take an hour). If the WCF service is idle long enough IIS recycles tha app pool aborting the worker threads. This problem has been circumvented by having the worker threads occasionally call a dummy service just to keep the app pool alive. If you think this whole setup is a really bad idea, I completely agree (not my code). So no need to comment that.
The problem is we still get an occasional ThreadAbortException. Is there any way to get additional information about what/who initiated the thread abort? I know it isn't our code.

IIS logs turned out to give the answer. AFAIK, if new binariers are loaded IIS waits until all service calls are finished (and no new call are accepted), then recycles the app pool. However, IIs has no knowledge of the background threads running after the service and therefore thinks it's free recycle the app pool. In some cases we've been uploading a new version while the background threads are still running. In any case, a very bad architecture.

How to keep quartz .net's scheduler alive?

I use quartz in my asp website, i initialize the scheduler in application_start method and shutdown in application_end method ,my trigger will fire everyday but I found that my scheduler will automatically shutdown if there are not request for a while ,so my background works will not triggered,are there any better way to keep the scheduler life long and only shutdown when the server stopped?

For better knowledge sharing:
There are two suggestions:
http://www.codeproject.com/Articles/12117/Simulate-a-Windows-Service-using-ASP-NET-to-run-sc
http://weblog.west-wind.com/posts/2007/May/10/Forcing-an-ASPNET-Application-to-stay-alive

In general, if you need reliable scheduling, you should not do it within a web site.
As you've found, the worker process will be shut down after a period of time. Even if you force the worker process to run all the time, there are conditions that may cause it to terminate as well. It's just not a good idea.
Instead, you should write a Windows Service and run quartz.net in that.
If you cannot install services (say you're in a shared hosting environment), then your options are more limited.

There is an IIS configuration that allows worker processes to stay on all the time. I found this setting through another SO answer link.
Edit C:\Windows\System32\inetsrv\config\applicationHost.config to include:
<applicationPools>
<add name="MyAppWorkerProcess" managedRuntimeVersion="v4.0" startMode="AlwaysRunning" />
</applicationPools>
Scott Guthrie (Microsoft Product Manager for .NET) has answerered a question directly related to the OP's question (link).
#Dominic Pettifer,
If I set startMode="AlwaysRunning" does this mean the web app will
'never' shut down and will always be kept running, even with no
traffic hitting the site for a long period (unless of course it's
manually shut down, or server is switched off/crashes etc.)? The
reason I ask is because I like to run background threads/services on
the IIS ASPNET worker process instead of using Windows Services (we
deal with clients with lots of security restrictions on their servers
which makes running a Windows Service difficalt or impossible).
Normally I have to devise something that hits the website periodically
to keep the ASPNET worker process alive and stop it from shutting
down.
This should mean that the application and worker process is always
running - so I think that does indeed handle your scenario well for
you.
Hope this helps,
Scott

I wondered the same thing. Ultimately, whilst I agree with the general consensus, I wanted to see how it could be done, because I've been in a similar situation myself, where Windows Services were not available to me.
All I did was create a new job which, when executed sends a HTTP request to the application itself. For me, I pointed it at a page which simply contained #Datetime.Now.ToString().
The action of sending a HTTP request to itself should be enough to keep the scheduler (and parent worker process) alive.
It does not however stop the application from being stopped/recycled without warning. If you wanted a way to handle that, then you'd likely need more than one site running which pings both itself and the other site. This way, if one site goes down, the other can hit it (assuming it's started) to bring it back.

A much simpler way to is use a quality assurance checker. Using the tool Zapix I was able to schedule my website to be quality checked every 20 minutes. Zapix simply visited the site and received and http response. By using Zapix, it mimicked the functionality of manually visiting the website to trigger the emails. That way, the Application Pool threads are constantly woke.