I have to refactor a fairly time-consuming process in one of my applications and after doing some research I think it's a perfect match for using TPL. I wanted to clarify my understanding of it and ask if there are any more issues which I should take into account.
In few words, I have a windows service, which runs overnight and sends out emails with data updates to around 10000 users. At presence, the whole process takes around 8 hrs to complete. I would like to reduce it to 2 hrs max.
Application workflow follows steps below:
1. Iterate through all users list
2. Check if this user has to be notified
3. If so, create an email body by calling external service
4. Send an email
Analysis of the code has shown that step 3 is the most time-consuming one and takes around 3,5 sec to complete. It means, that when processing 10000 users, my application waits well over 6 hrs in total for a response from the external service! I think this is a reason good enough to try to introduce some asynchronous and parallel processing.
So, my plan is to use Parallel class and ForEach method to iterate through users in step 1. As I can understand this should distribute processing each user into a separate thread, making them run in parallel? Processes are completely independent of each other and each doesn't return any value. In the case of any exception being thrown it will be persisted in logs db. As with regards to step 3, I would like to convert a call to external service into an async call. As I can understand this would release the resources on the thread so it could be reused by the Parallel class to start processing next user from the list?
I had a read through MS documentation regarding TPL, especially Potential Pitfalls in Data and Task Parallelism document and the only point I'm not sure about is "Avoid Writing to Shared Memory Locations". I am using a local integer to count a total number of emails processed. As with regards to all of the rest, I'm quite positive they're not applicable to my scenario.
My question is, without any implementation as yet. Is what I'm trying to achieve possible (especially the async await part for external service call)? Should I be aware of any other obstacles that might affect my implementation? Is there any better way of improving the workflow?
Just to clarify I'm using .Net v4.0
Yes, you can use the TPL for your problem. If you cannot influence your external problem, then this might be the best way.
However, you can make the best gains if you can get your external source to accept batches. Because this source could actually optimize the performance. Right now you have a message overhead of 10000 messages to serialize, send, work on, receive and deserialize. This is stuff that could be done once. In addition, your external source might be able to optimize the work they do if they know they will get multiple records.
So the bottom line is: if you need to optimize locally, the TPL is fine. If you want to optimize your whole process for actual gains, try to find out if your external source can help you, because that is where you can make some real progress.
You didn't show any code, and I'm assuming that step 4 (send an e-mail) is not that fast either.
With the presented case, unless your external service from step 3 (create an email body by calling external service) processes requests in parallel and supports a good load of simultaneous requests, you will not gain much with this refactor.
In other words, test the external service and the e-mail server first for:
Parallel request execution
The way to test this is to send at least 2 simultaneous requests and observe how long it takes to process them.
If it takes about double the time of a single, the requests have some serial processing, either they're queued or some broad lock is being taken.
Load test
Go up to 4, 8, 12, 16, 20, etc, and see where it starts to degrade.
You should set a limit on the amount of simultaneous requests to something that keeps execution time above e.g. 80% of the time it takes to process a single request, assuming you're the sole consumer
Or a few requests before it starts degrading (e.g. divide by the number of consumers) to leave the external service available for other consumers.
Only then can you decide if the refactor is worth. If you can't change the external service or the e-mail server, you must weight it they offer enough parallel capability without degrading.
Even so, be realistic. Don't let your service push the external service and the e-mail server to their limits in production.
Related
We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,
1)receives a request with input data from external clients
2)Will get 20 possibilities from database for one request based on input data after validation
3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data. Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.
4)After that Service will send back the all 20 solutions to the client.
This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds) considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.
So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.
I also tried async foreach in place of parallel.foreach as mentioned in below article but still performance is really bad.
https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239
As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.
I think service should not take more than 20 seconds for bulk request if it is lessa than 5 seconds for single request.
Can anyone please me what should be the way forward here to improve the performance ?
Thanks in advance.
Regards,
Raghu.
Looks like a lot of things happening here at the same time. I believe you have on nderlying issue that causes many side effects.
I will make the assumption that your server is sufficient in terms of CPU and memory to handle the concurrent connections (though the CPU 100% makes me wonder).
It seems to me that your problem, is that the parallel tasks (or threads), compete for the same resources. That would explain why multiple requests take much more time and why the async paradigm takes even more.
Let me explain:
The problem in practice
Parallel implementation: 1 or 2 request need minimum synchronization, so even if they compete for the same resources, it should be fine.
When 20 threads, try to access the same resources, a lot is happening and you come to a situation known as livelock.
When you switch to async, no requests await for a thread (they are waiting on the IO threads), so you make the problem even worse.
(I suspect that the problem is on your database. If your database server is the same machine, it would also explain the utilization).
The solution
Instead of trying to up the parallelism, find the contested resources and identify the problem.
If it's in your database (most probable scenario), then you need to identify the queries causing the trouble and fix them (indexes, statistics, query plans and whatnot). DB profilers showing locks and query execution plans are your friends for this.
If the problem is in your code, try to minimize the race conditions and imporve your algorithms.
To get a hint of where to look for, use the Visual Studio profiling tools: https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019 or any external .net profiling software.
We are running a Http Api and want to be able to set a limit to the number of requests a user can do per time unit. When this limit has been reached, we don't want the users to receive errors, such as Http 429. Instead we want to increase the response times. This has the result that the users can continue to work, but slower, and can then choose to upgrade or not upgrade its paying plan. This solution can quite easily be implemented using Thread.sleep (or something similar) for x number of seconds, on all requests of a user that has passed its limit.
We think that in worst case there might be a problem with the number of possible connections for a single server, since as long as we keep delaying the response, we keep a connection open, and therefore limiting the number of possible other connections.
All requests to the Api is running asynchronously. The Server itself is built to be scalable and is running behind a load balancer. We can start up additional servers if necessary.
When searching for this type of throttling, we find very few examples of this way of limiting the users, and the examples we found seemed not concerned at all about connections running out. So we wonder is this not a problem?
Are there any downsides to this that we are missing, or is this a feasible solution? How many connections can we have open simultaneously without starting to get problems? Can our vision be solved in another way, that is without giving errors to the user?
Thread.Sleep() is pretty much the worst possible thing you can do on a web server. It doesn't matter that you are running things asynchronously because that only applies to I/O bound operations and then frees the thread to do more work.
By using a Sleep() command, you will effectively be taking that thread out of commission for the time it sleeps.
ASP.Net App Pools have a limited number of threads available to them, and therefore in the worst case scenario, you will max out the total number of connections to your server at 40-50 (whatever the default is), if all of them are sleeping at once.
Secondly
This opens up a major attack vector in terms of DOS. If I am an attacker, I could easily take out your entire server by spinning up 100 or 1000 connections, all using the same API key. Using this approach, the server will dutifully start putting all the threads to sleep and then it's game over.
UPDATE
So you could use Task.Delay() in order to insert an arbitrary amount of latency in the response. Under the hood it uses a Timer which is much lighter weight than using a thread.
await Task.Delay(numberOfMilliseconds);
However...
This only takes care of one side of the equation. You still have an open connection to your server for the duration of the delay. Because this is a limited resource it still leaves you vulnerable to a DOS attack that wouldn't have normally existed.
This may be an acceptable risk for you, but you should at least be aware of the possibility.
Why not simply add a "Please Wait..." on the client to artificially look like it's processing? Adding artificial delays on server costs you, it leaves connections as well as threads tied up unnecessarily.
This is my first post here, so apologies if this isn't structured well.
We have been tasked to design a tool that will:
Read a file (of account IDs), CSV format
Download the account data file from the web for each account (by Id) (REST API)
Pass the file to a converter that will produce a report (financial predictions etc) [~20ms]
If the prediction threshold is within limits, run a parser to analyse the data [400ms]
Generate a report for the analysis above [80ms]
Upload all files generated to the web (REST API)
Now all those individual points are relatively easy to do. I'm interested in finding out how best to architect something to handle this and to do it fast & efficiently on our hardware.
We have to process roughly around 2 Million accounts. The square brackets gives an idea of how long each process takes on average. I'd like to use the maximum resources available on the machine - 24 core Xeon processors. It's not a memory intensive process.
Would using TPL and creating each of these as a task be a good idea? Each has to happen sequentially but many can be done at once. Unfortunately the parsers are not multi-threading aware and we don't have the source (it's essentially a black box for us).
My thoughts were something like this - assumes we're using TPL:
Load account data (essentially a CSV import or SQL SELECT)
For each Account (Id):
Download the data file for each account
ContinueWith using the data file, send to the converter
ContinueWith check threshold, send to parser
ContinueWith Generate Report
ContinueWith Upload outputs
Does that sound feasible or am I not understanding it correctly? Would it be better to break down the steps a different way?
I'm a bit unsure on how to handle issues with the parser throwing exceptions (it's very picky) or when we get failures uploading.
All this is going to be in a scheduled job that will run after-hours as a console application.
I would think about using some kind of messagebus. So you can seperate the steps and if one wouldn't work (for example because the REST Service isn't accessible for some time) you can store the message for processing them later on.
Depending on what you use as a messagebus you can introduce threads with it.
In my opinion you could better design workflows, handle exceptional states and so on, if you have a more high level abstraction like a service bus.
Also beaucase the parts could run indepdently they don't block each other.
One easy way could be to use servicestack messaging with Redis ServiceBus.
Some advantages quoted from there:
Message-based design allows for easier parallelization and introspection of computations
DLQ messages can be introspected, fixed and later replayed after server updates and rejoin normal message workflow
I think the easy way to start with multiple thread in your case, will be putting the entire operation for each account id in a thread (or better, in a ThreadPool). In the proposed way below, I think you will not need to control inter-thread operations.
Something like this to put the data on the thread pool queue:
var accountIds = new List<int>();
foreach (var accountId in accountIds)
{
ThreadPool.QueueUserWorkItem(ProcessAccount, accountId);
}
And this is the function you will process each account:
public static void ProcessAccount(object accountId)
{
// Download the data file for this account
// ContinueWith using the data file, send to the converter
// ContinueWith check threshold, send to parser
// ContinueWith Generate Report
// ContinueWith Upload outputs
}
We've built this app that needs to have some calculations done on a remote machine (actually a MatLab server). We're using web services to connect to the MatLab server and perform the calculations.
In order to speed things up, we've used Parallel.ForEach() in order to have multiple service calls going at the same time. If we're very conservative in setting ParallelOptions.MaxDegreeOfParallelism (DOP) to 4 or something, everything works fine and well.
However, if we let the framework decide on the DOP it will spawn so many threads that it forces the remote machine on its knees and timeouts start occurring ( > 10 minutes ).
How can we solve this issue? What I would LOVE to be able to do is use the response time to throttle the calls. If response time is less than 30 sec, keep adding threads, as soon as it's over 30 sec, use less. Any suggestions?
N.B. Related to the response in this question: https://stackoverflow.com/a/20192692/896697
Simplest way would be to tune for the best number of concurrent requests and hardcode that as you have done so far, however there are some nicer options if you are willing to put in some effort.
You could move from a Parallel.ForEach to using a thread pool. That way as things come back from the remote server you can either manually or programatically tune the number of available threads. reducing/increasing the number of available threads as things slow down/speed up, or even kill them if needed.
You could also do a variant of the above using Tasks which are the newer way of doing parallel/async stuff in .net.
Another option would be to use a timers and/or jobs model to schedule jobs every x milliseconds, which could then be throttled/relaxed as results returned from the server. The easiest way to get started would be using Quartz.Net.
I'm tasked to create a web application. I'm currently using c# & asp.net (mvc - but i doubt its relevant to the question) - am a rookie developer and somewhat new to .net.
Part of the logic in the application im building is to make requests to an external smsgateway by means of hitting a particular url with a request - either as part of a user-initiated action in the webapp (could be a couple of messages send) or as part of a scheduledtask run daily (could and will be several thousand message send).
In relation to a daily task, i am afraid that looping - say - 10.000 times in one thread (especially if im also to take action depending on the response of the request - like write to a db) is not the best strategy and that i could gain some performance/timesavings from some parallelization.
Ultimately i'm more afraid that thousands of users at the same time (very likely) will perform the action that triggers a request. With a naive implementation that spawns some kind of background thread (whatever its called) for each request i fear a scenario with hundreds/thousands of requests at once.
So if my assumptions are correct - how do i deal with this? do i have to manually spawn some appropriate number of new Thread()s and coordinate their work from a producer/consumer-like queue or is there some easy way?
Cheers
If you have to make 10,000 requests to a service then it means that the service's API is anemic - probably CRUD-based, designed as a thin wrapper over a database instead of an actual service.
A single "request" to a well-designed service should convey all of the information required to perform a single "unit of work" - in other words, those 10,000 requests could very likely be consolidated into one request, or at least a small handful of requests. This is especially important if requests are going to a remote server or may take a long time to complete (and 2-3 seconds is an extremely long time in computing).
If you do not have control over the service, if you do not have the ability to change the specification or the API - then I think you're going to find this very difficult. A single machine simply can't handle 10,000 outgoing connections at once; it will struggle with even a few hundred. You can try to parallelize this, but even if you achieve a tenfold increase in throughput, it's still going to take half an hour to complete, which is the kind of task you probably don't want running on a public-facing web site (but then, maybe you do, I don't know the specifics).
Perhaps you could be more specific about the environment, the architecture, and what it is you're trying to do?
In response to your update (possibly having thousands of users all performing an action at the same time that requires you to send one or two SMS messages for each):
This sounds like exactly the kind of scenario where you should be using Message Queuing. It's actually not too difficult to set up a solution using WCF. Some of the main reasons why one uses a message queue are:
There are a large number of messages to send;
The sending application cannot afford to send them synchronously or wait for any kind of response;
The messages must eventually be delivered.
And your requirements fit this like a glove. Since you're already on the Microsoft stack, I'd definitely recommend an asynchronous WCF service backed by MSMQ.
If you are working with SOAP, or some other type XML request, you may not have an issue dealing with the level of requests in a loop.
I set up something similar using a SOAP server with 4-5K requests with no problem...
A SOAP request to a web service (assuming .NET 2.0 and superior) looks something like this:
WebServiceProxyClient myclient = new WebServiceProxyClient();
myclient.SomeOperation(parameter1, parameter2);
myclient.Close();
I'm assuming that this code will will be embedded into your business logic that you will be trigger as part of the user initiated action, or as part of the scheduled task.
You don't need to do anything especial in your code to cope with a high volume of users. This will actually be a matter of scalling on your platform.
When you say 10.000 request, what do you mean? 10.000 request per second/minute/hour, this is your page hit per day, etc?
I'd also look into using an AsyncController, so that your site doesn't quickly become completely unusable.