We've built this app that needs to have some calculations done on a remote machine (actually a MatLab server). We're using web services to connect to the MatLab server and perform the calculations.
In order to speed things up, we've used Parallel.ForEach() in order to have multiple service calls going at the same time. If we're very conservative in setting ParallelOptions.MaxDegreeOfParallelism (DOP) to 4 or something, everything works fine and well.
However, if we let the framework decide on the DOP it will spawn so many threads that it forces the remote machine on its knees and timeouts start occurring ( > 10 minutes ).
How can we solve this issue? What I would LOVE to be able to do is use the response time to throttle the calls. If response time is less than 30 sec, keep adding threads, as soon as it's over 30 sec, use less. Any suggestions?
N.B. Related to the response in this question: https://stackoverflow.com/a/20192692/896697
Simplest way would be to tune for the best number of concurrent requests and hardcode that as you have done so far, however there are some nicer options if you are willing to put in some effort.
You could move from a Parallel.ForEach to using a thread pool. That way as things come back from the remote server you can either manually or programatically tune the number of available threads. reducing/increasing the number of available threads as things slow down/speed up, or even kill them if needed.
You could also do a variant of the above using Tasks which are the newer way of doing parallel/async stuff in .net.
Another option would be to use a timers and/or jobs model to schedule jobs every x milliseconds, which could then be throttled/relaxed as results returned from the server. The easiest way to get started would be using Quartz.Net.
Related
We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,
1)receives a request with input data from external clients
2)Will get 20 possibilities from database for one request based on input data after validation
3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data. Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.
4)After that Service will send back the all 20 solutions to the client.
This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds) considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.
So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.
I also tried async foreach in place of parallel.foreach as mentioned in below article but still performance is really bad.
https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239
As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.
I think service should not take more than 20 seconds for bulk request if it is lessa than 5 seconds for single request.
Can anyone please me what should be the way forward here to improve the performance ?
Thanks in advance.
Regards,
Raghu.
Looks like a lot of things happening here at the same time. I believe you have on nderlying issue that causes many side effects.
I will make the assumption that your server is sufficient in terms of CPU and memory to handle the concurrent connections (though the CPU 100% makes me wonder).
It seems to me that your problem, is that the parallel tasks (or threads), compete for the same resources. That would explain why multiple requests take much more time and why the async paradigm takes even more.
Let me explain:
The problem in practice
Parallel implementation: 1 or 2 request need minimum synchronization, so even if they compete for the same resources, it should be fine.
When 20 threads, try to access the same resources, a lot is happening and you come to a situation known as livelock.
When you switch to async, no requests await for a thread (they are waiting on the IO threads), so you make the problem even worse.
(I suspect that the problem is on your database. If your database server is the same machine, it would also explain the utilization).
The solution
Instead of trying to up the parallelism, find the contested resources and identify the problem.
If it's in your database (most probable scenario), then you need to identify the queries causing the trouble and fix them (indexes, statistics, query plans and whatnot). DB profilers showing locks and query execution plans are your friends for this.
If the problem is in your code, try to minimize the race conditions and imporve your algorithms.
To get a hint of where to look for, use the Visual Studio profiling tools: https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019 or any external .net profiling software.
I have to refactor a fairly time-consuming process in one of my applications and after doing some research I think it's a perfect match for using TPL. I wanted to clarify my understanding of it and ask if there are any more issues which I should take into account.
In few words, I have a windows service, which runs overnight and sends out emails with data updates to around 10000 users. At presence, the whole process takes around 8 hrs to complete. I would like to reduce it to 2 hrs max.
Application workflow follows steps below:
1. Iterate through all users list
2. Check if this user has to be notified
3. If so, create an email body by calling external service
4. Send an email
Analysis of the code has shown that step 3 is the most time-consuming one and takes around 3,5 sec to complete. It means, that when processing 10000 users, my application waits well over 6 hrs in total for a response from the external service! I think this is a reason good enough to try to introduce some asynchronous and parallel processing.
So, my plan is to use Parallel class and ForEach method to iterate through users in step 1. As I can understand this should distribute processing each user into a separate thread, making them run in parallel? Processes are completely independent of each other and each doesn't return any value. In the case of any exception being thrown it will be persisted in logs db. As with regards to step 3, I would like to convert a call to external service into an async call. As I can understand this would release the resources on the thread so it could be reused by the Parallel class to start processing next user from the list?
I had a read through MS documentation regarding TPL, especially Potential Pitfalls in Data and Task Parallelism document and the only point I'm not sure about is "Avoid Writing to Shared Memory Locations". I am using a local integer to count a total number of emails processed. As with regards to all of the rest, I'm quite positive they're not applicable to my scenario.
My question is, without any implementation as yet. Is what I'm trying to achieve possible (especially the async await part for external service call)? Should I be aware of any other obstacles that might affect my implementation? Is there any better way of improving the workflow?
Just to clarify I'm using .Net v4.0
Yes, you can use the TPL for your problem. If you cannot influence your external problem, then this might be the best way.
However, you can make the best gains if you can get your external source to accept batches. Because this source could actually optimize the performance. Right now you have a message overhead of 10000 messages to serialize, send, work on, receive and deserialize. This is stuff that could be done once. In addition, your external source might be able to optimize the work they do if they know they will get multiple records.
So the bottom line is: if you need to optimize locally, the TPL is fine. If you want to optimize your whole process for actual gains, try to find out if your external source can help you, because that is where you can make some real progress.
You didn't show any code, and I'm assuming that step 4 (send an e-mail) is not that fast either.
With the presented case, unless your external service from step 3 (create an email body by calling external service) processes requests in parallel and supports a good load of simultaneous requests, you will not gain much with this refactor.
In other words, test the external service and the e-mail server first for:
Parallel request execution
The way to test this is to send at least 2 simultaneous requests and observe how long it takes to process them.
If it takes about double the time of a single, the requests have some serial processing, either they're queued or some broad lock is being taken.
Load test
Go up to 4, 8, 12, 16, 20, etc, and see where it starts to degrade.
You should set a limit on the amount of simultaneous requests to something that keeps execution time above e.g. 80% of the time it takes to process a single request, assuming you're the sole consumer
Or a few requests before it starts degrading (e.g. divide by the number of consumers) to leave the external service available for other consumers.
Only then can you decide if the refactor is worth. If you can't change the external service or the e-mail server, you must weight it they offer enough parallel capability without degrading.
Even so, be realistic. Don't let your service push the external service and the e-mail server to their limits in production.
I have a Windows Service that has code similar to the following:
List<Buyer>() buyers = GetBuyers();
var results = new List<Result();
Parallel.Foreach(buyers, buyer =>
{
// do some prep work, log some data, etc.
// call out to an external service that can take up to 15 seconds each to return
results.Add(Bid(buyer));
}
// Parallel foreach must have completed by the time this code executes
foreach (var result in results)
{
// do some work
}
This is all fine and good and it works, but I think we're suffering from a scalability issue. We average 20-30 inbound connections per minute and each of those connections fire this code. The "buyers" collection for each of those inbound connections can have from 1-15 buyers in it. Occasionally our inbound connection count sees a spike to 100+ connections per minute and our server grinds to a halt.
CPU usage is only around 50% on each server (two load balanced 8 core servers) but the thread count continues to rise (spiking up to 350 threads on the process) and our response time for each inbound connection goes from 3-4 seconds to 1.5-2 minutes.
I suspect the above code is responsible for our scalability problems. Given this usage scenario (parallelism for I/O operations) on a Windows Service (no UI), is Parallel.ForEach the best approach? I don't have a lot of experience with async programming and am looking forward to using this opportunity to learn more about it, figured I'd start here to get some community advice to supplement what I've been able to find on Google.
Parallel.Foreach has a terrible design flaw. It is prone to consume all available thread-pool resources over time. The number of threads that it will spawn is literally unlimited. You can get up to 2 new ones per second driven by heuristics that nobody understands. The CoreCLR has a hill climbing algorithm built into it that just doesn't work.
call out to an external service
Probably, you should find out what's the right degree of parallelism calling that service. You need to find out by testing different amounts.
Then, you need to restrict Parallel.Foreach to only spawn as many threads as you want at a maximum. You can do that using a fixed concurrency TaskScheduler.
Or, you change this to use async IO and use SemaphoreSlim.WaitAsync. That way no threads are blocked. The pool exhaustion is solved by that and the overloading of the external service as well.
We are running a Http Api and want to be able to set a limit to the number of requests a user can do per time unit. When this limit has been reached, we don't want the users to receive errors, such as Http 429. Instead we want to increase the response times. This has the result that the users can continue to work, but slower, and can then choose to upgrade or not upgrade its paying plan. This solution can quite easily be implemented using Thread.sleep (or something similar) for x number of seconds, on all requests of a user that has passed its limit.
We think that in worst case there might be a problem with the number of possible connections for a single server, since as long as we keep delaying the response, we keep a connection open, and therefore limiting the number of possible other connections.
All requests to the Api is running asynchronously. The Server itself is built to be scalable and is running behind a load balancer. We can start up additional servers if necessary.
When searching for this type of throttling, we find very few examples of this way of limiting the users, and the examples we found seemed not concerned at all about connections running out. So we wonder is this not a problem?
Are there any downsides to this that we are missing, or is this a feasible solution? How many connections can we have open simultaneously without starting to get problems? Can our vision be solved in another way, that is without giving errors to the user?
Thread.Sleep() is pretty much the worst possible thing you can do on a web server. It doesn't matter that you are running things asynchronously because that only applies to I/O bound operations and then frees the thread to do more work.
By using a Sleep() command, you will effectively be taking that thread out of commission for the time it sleeps.
ASP.Net App Pools have a limited number of threads available to them, and therefore in the worst case scenario, you will max out the total number of connections to your server at 40-50 (whatever the default is), if all of them are sleeping at once.
Secondly
This opens up a major attack vector in terms of DOS. If I am an attacker, I could easily take out your entire server by spinning up 100 or 1000 connections, all using the same API key. Using this approach, the server will dutifully start putting all the threads to sleep and then it's game over.
UPDATE
So you could use Task.Delay() in order to insert an arbitrary amount of latency in the response. Under the hood it uses a Timer which is much lighter weight than using a thread.
await Task.Delay(numberOfMilliseconds);
However...
This only takes care of one side of the equation. You still have an open connection to your server for the duration of the delay. Because this is a limited resource it still leaves you vulnerable to a DOS attack that wouldn't have normally existed.
This may be an acceptable risk for you, but you should at least be aware of the possibility.
Why not simply add a "Please Wait..." on the client to artificially look like it's processing? Adding artificial delays on server costs you, it leaves connections as well as threads tied up unnecessarily.
I have been given a windows service written by a previous intern at my current internship that monitors an archive and alerts specific people through emails and pop-ups should one of the recorded values go outside a certain range. It currently uses a timer to check the archive every 30 seconds, and I have been asked if I would be able to update it to allow a choice of time depending on what "tag" is being monitored. It uses an XML file to keep track of which tags are being monitored. Would creating multiple timers in the service be the most efficient way of going about this? I'm not really sure what approach to take.
The service is written in C# using .NET 3.5.
Depending on the granularity, you could use a single timer that is a common factor of the timing intervals they want. Say they want to put in the XML file that each archive is to be checked every so many minutes. You set up a timer that goes off once a minute, and you check how long it's been since you did each one and whether to do it or not.
If you're getting a chance to re-architect, I would move away from a service to a set of scheduled tasks. Write it so one task does one archive. Then write a controller program that sets up the scheduled tasks (and can stop them, change them etc.) The API for scheduled tasks on Windows 7 is nice and understandable, and unlike a service you can impose restrictions like "don't do it if the computer is on battery" or "only do it if the machine is idle" along with your preferences for what to do if a chance to run the task was missed. 7 or 8 scheduled tasks, each on their own schedule, using the same API of yours, passing in the archive path and the email address, is a lot neater than one service trying to juggle everything at once. Plus the machine will start up faster when you don't have yet another autostart service on it.
Efficient? Possibly not - especially if you have lots of tags, as each timer takes a tiny but finite amount of resources.
An alternative approach might be to have one timer that fires every second, and when that happens you check a list of outstanding requests.
This has the benefit of being easier to debug if things go wrong as there's only one active thread.
As in most code maintenance situations, however, it depends on your existing code, your ability, and how you feel more comfortable.
I woould suggest to just use one timer scheduled at the least common divisor.
For example configure your timer to signal every second and you can handle every interval (1 second, 2 seconds, ...) by counting the according number of timer ticks.