I know there are many cases which are good cases to use multi-thread in an application, but when is it the best to multi-thread a .net web application?
A web application is almost certainly already multi threaded by the hosting environment (IIS etc). If your page is CPU-bound (and want to use multiple cores), then arguably multiple threads is a bad idea, as when your system is under load you are already using them.
The time it might help is when you are IO bound; for example, you have a web-page that needs to talk to 3 external web-services, talk to a database, and write a file (all unrelated). You can do those in parallel on different threads (ideally using the inbuilt async operations, to maximise completion-port usage) to reduce the overall processing time - all without impacting local CPU overly much (here the real delay is on the network).
Of course, in such cases you might also do better by simply queuing the work in the web application, and having a separate service dequeue and process them - but then you can't provide an immediate response to the caller (they'd need to check back later to verify completion etc).
IMHO you should avoid the use of multithread in a web based application.
maybe a multithreaded application could increase the performance in a standard app (with the right design), but in a web application you may want to keep a high throughput instead of speed.
but if you have a few concurrent connection maybe you can use parallel thread without a global performance degradation
Multithreading is a technique to provide a single process with more processing time to allow it to run faster. It has more threads thus it eats more CPU cycles. (From multiple CPU's, if you have any.) For a desktop application, this makes a lot of sense. But granting more CPU cycles to a web user would take away the same cycles from the 99 other users who are doing requests at the same time! So technically, it's a bad thing.
However, a web application might use other services and processes that are using multiple threads. Databases, for example, won't create a separate thread for every user that connects to them. They limit the number of threads to just a few, adding connections to a connection pool for faster usage. As long as there are connections available or pooled, the user will have database access. When the database runs out of connections, the user will have to wait.
So, basically, the use of multiple threads could be used for web applications to reduce the number of active users at a specific moment! It allows the system to share resources with multiple users without overloading the resource. Instead, users will just have to stand in line before it's their turn.
This would not be multi-threading in the web application itself, but multi-threading in a service that is consumed by the web application. In this case, it's used as a limitation by only allowing a small amount of threads to be active.
In order to benefit from multithreading your application has to do a significant amount of work that can be run in parallel. If this is not the case, the overhead of multithreading may very well top the benefits.
In my experience most web applications consist of a number of short running methods, so apart from the parallelism already offered by the hosting environment, I would say that it is rare to benefit from multithreading within the individual parts of a web application. There are probably examples where it will offer a benefit, but my guess is that it isn't very common.
ASP.NET is already capable of spawning several threads for processing several requests in parallel, so for simple request processing there is rarely a case when you would need to manually spawn another thread. However, there are a few uncommon scenarios that I have come across which warranted the creation of another thread:
If there is some operation that might take a while and can run in parallel with the rest of the page processing, you might spawn a secondary thread there. For example, if there was a webservice that you had to poll as a result of the request, you might spawn another thread in Page_Init, and check for results in Page_PreRender (waiting if necessary). Though it's still a question if this would be a performance benefit or not - spawning a thread isn't cheap and the time between a typical Page_Init and Page_Prerender is measured in milliseconds anyway. Keeping a thread pool for this might be a little bit more efficient, and ASP.NET also has something called "asynchronous pages" that might be even better suited for this need.
If there is a pool of resources that you wish to clean up periodically. For example, imagine that you are using some weird DBMS that comes with limited .NET bindings, but there is no pooling support (this was my case). In that case you might want to implement the DB connection pool yourself, and this would necessitate a "cleaner thread" which would wake up, say, once a minute and check if there are connections that have not been used for a long while (and thus can be closed off).
Another thing to keep in mind when implementing your own threads in ASP.NET - ASP.NET likes to kill off its processes if they have been inactive for a while. Thus you should not rely on your thread staying alive forever. It might get terminated at any moment and you better be ready for it.
Related
Everybody knows that asynchrony gives you "better throughput", "scalability", and more efficient in terms of resources consumption. I also thought this (simplistic) way before doing an experiment below. It basically tells that if we take into account all the overhead for asynchronous code and compare it against properly configured synchronous code it yields little to no performance/throughput/resource consumption advantages.
The question: Does asynchronous code actually perform so much better comparing to the synchronous code with correctly configured thread pool? May be my performance tests are flawed in some dramatic way?
Test setup: Two ASP.NET Web API methods with JMeter trying to call them with 200 threads thread group (30 seconds rump up time).
[HttpGet]
[Route("async")]
public async Task<string> AsyncTest()
{
await Task.Delay(_delayMs);
return "ok";
}
[HttpGet]
[Route("sync")]
public string SyncTest()
{
Thread.Sleep(_delayMs);
return "ok";
}
Here is response time (log scale). Notice how synchronous code becomes faster when Thread Pool injected enough threads. If we were to set up Thread Pool beforehand (via SetMinThreads) it would outperform async right from the start.
What about resources consumption you would ask. "Thread has big cost in terms of CPU time scheduling, context switching and RAM footprint". Not so fast. Threads scheduling and context switching is efficient. As far as the stack usage goes thread does not instantly consume the RAM, but rather just reserve virtual address space and commit only a tiny fraction which is actually needed.
Let's look at what the data says. Even with bigger amount threads sync version has smaller memory footprint (working set which maps into the physical memory).
UPDATE. I want to post the results of follow-up experiment which should be more representational since avoids some biases of the first one.
First of all, the results of the first experiment are taken using IIS Express, which is basically dev time server, so I needed to move away from that. Also, considering the feedback I've isolated load generation machine from the server (two Azure VMs in the same network). I've also discovered that some IIS threading limits are from hard to impossible to violate and ended up switching to ASP.NET WebAPI self-hosting to eliminate IIS from the variables as well. Note that memory footprints/CPU times are radically different with this test, please do not compare numbers across the different test runs as setups are totally different (hosting, hardware, machines setup). Additionally, when I moved to another machines and another hosting solution the Thread Pool strategy changed (it is dynamic) and injection rate increased.
Settings: Delay 100ms, 200 JMeter "users", 30 sec ramp-up time.
I want to conclude these experiments with the following: Yes, under some specific
(more laboratory like) circumstances it's possible to get comparable results for sync vs. async, but in real world cases where workload can not be 100% predictable and workload is uneven we inevitably will hit some kind of threading limits: either server side limits, or Thread Pool grow limits (and bear in mind that thread pool management is automatic mechanism with not always easily predictable properties). Additionally, sync version does have a bigger memory footprint (both working set, and way bigger virtual memory size). As far as CPU consumption is concerned async also wins (CPU time per request metric).
On IIS with default settings the situation is even more dramatic: synchronous version is order(s) of magnitude slower (and smaller throughput) due to quite tight limit on threads count - 20 per CPU.
PS. Do use asynchronous pipelines for IO! [... sigh of relief...]
Everybody knows that asynchrony gives you "better throughput", "scalability", and more efficient in terms of resources consumption.
Scalability, yes. Throughput: it depends. Each asynchronous request is slower than the equivalent synchronous request, so you would only see a throughput benefit when scalability comes into play (i.e., there are more requests than threads available).
Does asynchronous code actually perform so much better comparing to the synchronous code with correctly configured thread pool?
Well, the catch there is "correctly configured thread pool". What you're assuming is that you can 1) predict your load, and 2) have a server big enough to handle it using one thread per request. For many (most?) real-world production scenarios, either or both of these are not true.
From my article on async ASP.NET:
Why not just increase the size of the thread pool [instead of using async]? The answer is twofold: Asynchronous code scales both further and faster than blocking thread pool threads.
First, asynchronous code scales further than synchronous code. With more realistic example code, the total scalability of ASP.NET servers (stress tested) showed a multiplicative increase. In other words, an asynchronous server could handle several times the number of continuous requests as a synchronous server (with both thread pools turned up to the maximum for that hardware). However, these experiments (not done by me) were done on a expected "realistic baseline" for average ASP.NET apps. I don't how the same results would carry over to a noop string return.
Second, asynchronous code scales faster than synchronous code. This one is pretty obvious; synchronous code scales fine up to the number of thread pool threads, but then can't scale faster than the thread injection rate. So you get that really slow response to a sudden heavy load, as shown in the beginning of your response time graph.
I think the work you've done is interesting; I am particularly surprised at the memory usage differences (or rather, lack of difference). I'd love to see you work this into a blog post. Recommendations:
Use ASP.NET Core for your tests. The old ASP.NET had only a partially-asynchronous pipeline; ASP.NET Core would be necessary for a more "pure" comparison of sync vs async.
Don't test locally; there are a lot of caveats when doing that. I'd recommend choosing a VM size (or single-instance Docker container or whatever) and testing in the cloud for repeatability.
Also try stress testing in addition to load testing. Keep increasing load until the server is totally overwhelmed, and see how both the async and sync servers respond.
As a final reminder (also from my article):
Bear in mind that asynchronous code does not replace the thread pool. This isn’t thread pool or asynchronous code; it’s thread pool and asynchronous code. Asynchronous code allows your application to make optimum use of the thread pool. It takes the existing thread pool and turns it up to 11.
Trully asynchronous code (I/O) is more scalable because it releases thread pool threads for other work instead of blocking them. So, for the same number of threads being, it can handle more requests.
But it does that at the cost of more control data structures and more work. So, (other than saving thread pool threads) it consumes more resources (memory, CPU).
It's all about availability, not performance.
This has been bugging me for a while. Assuming that I am not using any explicit form of task parallelism (e.g. Paralell.ForEach), how many threads are natively used when I deploy a web application to a Windows Server?
In other words, if the web server has an eight-core processor, does the application only use one of those cores if I am not explicitly telling it to use more?
I'll bet I have missed something simple here, so flame on--but I still want to know the answer. Thanks.
First, you have to consider that a web server is, by its nature, a multi-threaded system because it has to be able to respond to multiple requests at the same time (there are ways to do this in a single threaded environment, but it's not very efficient, and IIS does not support it). If two users access the site at the same time, two separate threads are servicing each request in most cases.
Once you get into scalability issues, and async support, then a thread can actually service multiple requests (or rather, a request can be put on a queue and the IIS worker threads can be reused to process other requests).
So you are looking at this from the aspect of a single user. And that single user will likely run only a single thread of activity during that web request (ignoring async, io completion ports, etc..). i.e. you can think of a single web request as being equal to a single thread in the general sense (but there are so many things that can spin off other threads, that this isn't something that you can really count on).
When you use something like Parallel.ForEach it's so that your single users request can now execute things in multiple threads to improve efficiency.
An IIS web site has a worker process associated with it. That worker process can be configured and tuned to control how many threads it uses. But just remember, that your web site is running under the control of IIS, not as it's own application.
The topic of threads can get really confusing as you move along .NET versions.
But to over-simplify, it depends on your processModel configuration for ASP.NET. (machine.config) autoConfig=true/false etc.
at a bare minimum, in .NET 4.0, the min/max WorkerThreads and min/max IOCompletionThreads determine the amount of threads in play.
e.g. If the minWorkerThreads = 1 (default value), then the total minThreads = 1 * no:of cores..
same with maxWorkerThreads.
Based on the load, the worker process can ramp up the threads needed to service the requests, using an undisclosed algorith.. minWorkerThreads to maxWorkerThreads. (of course based on availability of ThreadPool threads etc.)
I have a console application(c#) where I have to call various third party API's and collect data. This I have to do simultaneously for different users. I am using threads for it. But as the number of users are increasing this service is eating into the CPU performance. It is affecting other processes. Is there a way we can use threads for parallel processing but do not affect the CPU performance in a huge way.
I assume from your question that you're creating threads manually, and so the quick way to answer this is to suggest that you use an API like the Task Parallel Library, because this will take an arbitrary number of tasks and try to use a sensible number of threads to process them - so given 500 API requests, it would limit itself to just a few threads.
However, to answer in more detail: the typical reason that you would see this problem is that code is creating too many threads. Threads are not free resources - they are expensive.
A made up example based on your question might be this:
you have 5 3rd party APIs that you need to call, and each is going to return ~1MB of data per user
you call each API on a separate background thread, for each user
you have 100 users
you therefore have created 500 threads in total, each of which is waiting on data from the network
The problem here is that there are 500 threads the program is trying to manage, and they are all waiting on the slowest piece of the system - the network.
More simply, we are trying to download 500 pieces of data at once (which in this example would mean everything finishes slowly), rather than downloading them one at a time so that individual items will finish earlier. Because each thread will be doing nothing (just waiting for the network), the CPU will switch between idle threads continually. As you increase your number of users, the number of threads increases - which increases the CPU usage just for switch between threads, even though each thread is actually downloading more slowly. This is (approximately) why you'll be seeing slower performance as your user count goes up.
A better example would be to take the same scenario and use just one background thread:
you have 5 3rd party APIs that you need to call, and each is going to return ~1MB of data per user
each API call is put into a queue and the queue is processed by a single thread
you have 100 users
you therefore have 1 thread running in the background which is using the full available bandwidth of the network for each request
In this example, your CPU usage will be pretty consistent - no matter how many users you have, there is only one background thread running, so context switching is minimised. Each individual API call runs at the maximum rate of the network card and so finishes as quickly as possible.
The reality is that one thread is probably not enough: a single request is unlikely to saturate the network, as there will be limiting factors elsewhere. But this is something you can tune later: maybe 2 or 3 threads would be more performant, but 4 threads would be slower again. The general rule when threading is to start small and work up, not to create a thread for each piece of work.
First, run a profiler and checkout some refactoring tools to see if you can perform code optimization to resolve the issue. If your application is still overloading the server then setup or purchase load balancing. In the meantime, if you are running the latest OS's you could try setting a hacky CPU rate limit...however, that may not work for the needs you described.
I have a basic queue of tasks executing (c# WinForms App that talks to 3 separate systems). Everything is great until one of the web services decides not to respond in with the usual speed.
I'm not interested in speeding things up via multi threading the jobs, but now that I am using it in production, I can see the benefit of having at least two threads running jobs - if one blocks and it's an anomaly, the other will keep truckin', and if both block, then probably any number of threads would and I just deal with it.
So, the question is: is this a common pattern I just described, and is there a name for that pattern and/or some awesome reference or framework, or anything to help me not re-invent any wheels.
Additions based on Comments/Answers
The tasks can be run simultaneously. I have chosen not to peruse a multi-threaded design for the purposes of speed, but I am now considering it to attain consistent performance in the face of infrequent task latency.
My assumption is that every once in a while a call to a web service takes disproportionately longer to complete while still being considered non-exceptional. This has a NON(corrected)-negligible impact on the total run time of, say, N jobs, if the average execution time is 1 second (including a host of disparate web service calls) and 0.0001% of the time a web service takes 15 seconds to respond.
Is a thread pool just another way of saying, "spin up worker threads and manage their state manually" ? Or is there something that can help me manage the complexity? I worry that the chance of introducing bug(s) grows out of proportion to the benefits in this case...
I think I am looking for something similar to a thread pool, but one that only spins up additional threads when latency is detected.
If anyone can give me more info on what one of the comments refers to as a work-stealing thread, that sounds promising.
The reason I didn't use the BackgroundWorker component is because they seem to be built for the case when you know how many workers you want, and I'd ideally keep the design flexible
PS: Thanks again.
Thanks!
It depends on how important is the order of the queue items and how important it is that an item is completed before the next one is processed.
If one item must be completely processed before the next one is, then basically you are stuck.
If not, you may decide to implement a simple pool of worker threads. if .NET 4.0 is an option I would recommend using the Parallel Extensions for that and especially the AsParallel() and AsOrdered() methods.
It sounds like what you might be looking for here is the Backgroundworker. A class that neatly encapsulates launching a worker thread, monitoring progress and receiving results. http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx. It's quick and easy to use just wire up the DoWork, ProgressChanged and RunWorkerCompleted events and then start it.
The producer-consumer pattern may be the best fit for this, and using a queue (either Queue<T> wrapped in a lock or the new ConcurrentQueue<T>) is a good approach. It also gives you a place to recycle web service requests that fail due to timeouts or dropped connections.
If you want to use more than the maximum default of two simultaneous web connections then add this to your app.config (replace "10" with your new maximum):
<configuration>
<system.net>
<connectionManagement>
<add address="*" maxconnection="10"/>
</connectionManagement>
</system.net>
</configuration>
If you're using .Net 4 you can also have a look at Tasks
I’m looking for the best way of using threads considering scalability and performance.
In my site I have two scenarios that need threading:
UI trigger: for example the user clicks a button, the server should read data from the DB and send some emails. Those actions take time and I don’t want the user request getting delayed. This scenario happens very frequently.
Background service: when the app starts it trigger a thread that run every 10 min, read from the DB and send emails.
The solutions I found:
A. Use thread pool - BeginInvoke:
This is what I use today for both scenarios.
It works fine, but it uses the same threads that serve the pages, so I think I may run into scalability issues, can this become a problem?
B. No use of the pool – ThreadStart:
I know starting a new thread takes more resources then using a thread pool.
Can this approach work better for my scenarios?
What is the best way to reuse the opened threads?
C. Custom thread pool:
Because my scenarios occurs frequently maybe the best way is to start a new thread pool?
Thanks.
I would personally put this into a different service. Make your UI action write to the database, and have a separate service which either polls the database or reacts to a trigger, and sends the emails at that point.
By separating it into a different service, you don't need to worry about AppDomain recycling etc - and you can put it on an entire different server if and when you want to. I think it'll give you a more flexible solution.
I do this kind of thing by calling a webservice, which then calls a method using a delegate asynchronously. The original webservice call returns a Guid to allow tracking of the processing.
For the first scenario use ASP.NET Asynchronous Pages. Async Pages are very good choice when it comes to scalability, because during async execution HTTP request thread is released and can be re-used.
I agree with Jon Skeet, that for second scenario you should use separate service - windows service is a good choice here.
Out of your three solutions, don't use BeginInvoke. As you said, it will have a negative impact on scalability.
Between the other two, if the tasks are truly background and the user isn't waiting for a response, then a single, permanent thread should do the job. A thread pool makes more sense when you have multiple tasks that should be executing in parallel.
However, keep in mind that web servers sometimes crash, AppPools recycle, etc. So if any of the queued work needs to be reliably executed, then moving it out of process is a probably a better idea (such as into a Windows Service). One way of doing that, which preserves the order of requests and maintains persistence, is to use Service Broker. You write the request to a Service Broker queue from your web tier (with an async request), and then read those messages from a service running on the same machine or a different one. You can also scale nicely that way by simply adding more instances of the service (or more threads in it).
In case it helps, I walk through using both a background thread and Service Broker in detail in my book, including code examples: Ultra-Fast ASP.NET.