We Keep Coding

Using thousands of Tasks with a timeout efficiently

I am implementing a Library L that communicates via Sockets with another application A.
Basic workflow is as followed:
L connects to A.
L sends ~50.000 pieces of information I to A, and
creates a task T for every I that is sent out.
L listens for incoming results from A, and once reuslts are there, uses a
TaskCompletionSource to set the results of the Tasks T
L creates a Task T2 with a set Timeout (Task.WhenAny(T,Task.Delay(xx))
L uses Task.WhenAll(T2) to wait for timeout or results on all sent information.
Managing the underlying data structure is no problem at all. The main problem is that assembling the "main" Task.WhenAll(T2) costs around 5-6 seconds on my computer with ca. 50.000 entries (creating 50.000*2+1 tasks).
I can't think off a more lightweight way that accomplishes the same, however. It should use all Cores available and be non-blocking, and support timeouts aswell.
Is there a way to accomplish the same using the Parallel- or ThreadPool classes which enhances the performance?
EDIT:
Code showing how the basic setup is:
https://dotnetfiddle.net/gIq2DP

Start a total of n LongRunningTasks, where n is the number of cores on your machine. Each task should run on one core. It would be a waste to create 50K new tasks for every I that you want to send. Instead design the tasks to accept I and the socket information - where this information is to be sent.
Create a BlockingCollection<Tuple<I, SocketInfo>>. Start one task to populate this blocking collection. The other n long running tasks that you created earlier can keep taking tuples of information and the address to send the information and then perform the job for you in a loop that will break when blocking collection is done.
Timeouts can be set in the long running tasks itself.
This entire setup will keep your CPU busy to the maximum with useful work rather than keeping it needlessly busy with a "job" of 50K tasks' creation.
Since the operations (like this network operation) which happen beyond the main memory are very very slow for the CPU, feel free to set n not just equal to number of cores in your machine but even thrice that value. In my code demonstration I have set it equal to the number of cores only.
With the code at the provided link, this is one way...
using System;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Linq.Expressions;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
namespace TestConsoleApplication
{
public static class Test
{
public static void Main()
{
TaskRunningTest();
}
private static void TaskRunningTest()
{
var s = new Stopwatch();
const int totalInformationChunks = 50000;
var baseProcessorTaskArray = new Task[Environment.ProcessorCount];
var taskFactory = new TaskFactory(TaskCreationOptions.LongRunning, TaskContinuationOptions.None);
var tcs = new TaskCompletionSource<int>();
var itemsToProcess = new BlockingCollection<Tuple<Information, Address>>(totalInformationChunks);
s.Start();
//Start a new task to populate the "itemsToProcess"
taskFactory.StartNew(() =>
{
// Add Tuples of Information and Address to which this information is to be sent to.
Console.WriteLine("Done intializing all the jobs...");
// Finally signal that you are done by saying..
itemsToProcess.CompleteAdding();
});
//Initializing the base tasks
for (var index = 0; index < baseProcessorTaskArray.Length; index++)
{
var thisIndex = index;
baseProcessorTaskArray[index] = taskFactory.StartNew(() =>
{
while (!itemsToProcess.IsAddingCompleted && itemsToProcess.Count != 0)
{
Tuple<Information, Address> item;
itemsToProcess.TryTake(out item);
//Process the item
tcs.TrySetResult(thisIndex);
}
});
}
// Need to provide new timeout logic now
// Depending upon what you are trying to achieve with timeout, you can devise out the way
// Wait for the base tasks to completely empty OR
// timeout and then stop the stopwatch.
Task.WaitAll(baseProcessorTaskArray);
s.Stop();
Console.WriteLine(s.ElapsedMilliseconds);
}
private class Address
{
//This class should have the socket information
}
private class Information
{
//This class will have the Information to send
}
}
}

Profiling shows that most time (90%?) is spent in timer setup, expiration and disposal. This seems plausible to me.
Maybe you can create your own super cheap timeout mechanism. Enqueue timeouts into a priority queue ordered by expiration time. Then, run a single timer every 100ms and make that timer expire everything in the priority queue that is due.
The cost of doing this would be one TaskCompletionSource per timeout and some small further processing.
You can even cancel timeouts by removing them from the queue and just dropping the TaskCompletionSource.

Windows Phone, Multiple HTTP request parallel, how many?

In my Windows Phone 8 app, Im fetching list of items from web api. After that I loop all items and get details for each Item.
Right now my code is something like this:
List<plane> planes = await planeService.getPlanes(); // Get all planes from web api
foreach(Plane plane in planes)
{
var details = await planeService.getDetails(plane.id); // Get one plane details from web api
drawDetails(details);
}
How can I improve this to make multiple request in parallel and what is resonable number of request running parallel? The planes list can be anything from 0 to 100 objects, typically max 20.

How can I improve this to make multiple request in parallel?
You can do the parallel processing like below (untested). It uses SemaphoreSlim to throttle getDetails requests.
async Task ProcessPlanes()
{
const int MAX_REQUESTS = 50;
List<plane> planes = await planeService.getPlanes(); // Get all planes from web api
var semaphore = new SemaphoreSlim(MAX_REQUESTS);
Func<string, Task<Details>> getDetailsAsync = async (id) =>
{
await semaphore.WaitAsync();
try
{
var details = await planeService.getDetails(id);
drawDetails(details);
return details;
}
finally
{
semaphore.Release();
}
};
var tasks = planes.Select((plane) =>
getDetailsAsync(plane.id));
await Task.WhenAll(tasks.ToArray());
}
what is resonable number of request running parallel? The planes list
can be anything from 0 to 100 objects, typically max 20.
It largely dependents on the server, but I don't think there's an ultimate answer to this. For example, check this question:
A reasonable number of simultaneous, asynchronous ajax requests
As far as the WP8 client goes, I believe it can spawn 100 parallel requests without a problem.

I don't know what the limit is for network connections, but there will be one.
If there wasn't, the only problem would be the amount of memory used to keep that many requests alive.
So, assuming the underlying operating system will handle throttling properly, I would do something this:
List<plane> planes = await planeService.getPlanes();
var allDetails = Task.WhenAll(from plane in plains
select planeService.getDetails(plane.id));
foreach(var details in allDetails)
{
drawDetails(details);
}
NOTE: You should follow common naming conventions to help others understand your code. Asynchronous methods should be suffixed Async and, in *C#, method names are always CamelCase.

You should check the ServicePoint, this will provides connection management for HTTP connections. The default maximum number of concurrent connections allowed by a ServicePoint object is 2. So if you need to increase it you can use ServicePointManager.DefaultConnectionLimit property. Just check the link in MSDN there you can see a sample. And set the value you need. This might help you..

Program doesn't use all hardware resources

I'm working on one program that takes information from files and then stores them in MySQL database. This MySQL database is located in another dedicated server which is much more powerful than this server here. Data is being sent over LAN using 1gbps connection.
It is using 8 threads because my server has 8 cores, but somehow it runs so slowly.
CPU is: Intel Xeon E3-1270 v 3 # 3.50Ghz
RAM: 16 GB ECC
HDD: SATA 3 1TB
My program's CPU usage is only 0-5%
CPU affinity is all 8 cores
So, do you have any ideas what's wrong or how can I increase the speed of my program?
UPDATE:
I updated my code and it appears to be faster:
Parallel.For(0, this.data_files.Count, new ParallelOptions { MaxDegreeOfParallelism = this.MaxThreads }, i =>
{
this.ThreadCount++;
this.ParseFile(this.GetSource());
});
Here's a code snippet that deploys threads:
while (true)
{
if (this.ThreadCount < this.MaxThreads)
{
Task.Factory.StartNew(() =>
this.ParseFile(this.GetFile())
);
this.ThreadCount++;
}
else
{
Thread.Sleep(1);
}
this.UpdateConsole();
}
GetFile function:
private string GetFile()
{
string file = "";
string source = "";
while (true)
{
if (this.data_files.Count() != 0)
{
file = this.data_files[0];
this.data_files.RemoveAt(0);
if (File.Exists(file) == true)
{
source = File.ReadAllText(file);
File.Delete(file);
break;
}
}
}
return source;
}

I'm working on one program that takes information from files and then stores them in MySQL database.
Clearly your program is not CPU bound, it's IO bound. The bottlenecks are going to be based on your hard disk(s) and your network connection. Odds are even a single thread is going to be able to ensure proper utilization of these resources (in a well designed application). Adding extra threads generally won't help, it'll just create a bunch of threads that will spend their time waiting on various IO operations.

To use all the hardware resources is not the right goal for a program to have.
Instead, a better goal is to be as fast as possible. This is significantly different. While using more hardware resources can help, it is not always sufficient.
Sometimes, adding more resources to a problem doesn't help. In those cases, don't. Adding threads makes your program more complex, but not necessarily faster as you've seen.
C# already has good Asynchronous programming features with the TPL (which you are already using), so why not take advantage of that?
This will mean that the .NET framework will automatically manage the threads for you in an efficient way.
Here's what I propose:
foreach (var file in GetFilesToRead()) {
var task = PerformOperation(file);
// Keep a list of tasks, if you wish.
}
...
Task PerformOperation (string filename) {
var file = await ReadFile(file);
await ParseFile(file);
DoSomething();
}
Note that even in CPU-bound programs, threads (and tasks) may not help you if you're using locks.
Although locks help keep programs well-behaved, they come at a significant performance cost.
Within a lock, only one thread may be executing at a time.
This means that the first thread is locking your _lock instance, and then the other threads are waiting for that lock to be released.
In your program, only one thread is active at a time.
To solve this, don't use locks. Instead, write programs that do not need locks at all. Copy variables instead of sharing them. Use immutable collections instead of mutable collections and so on.
My program above uses exactly zero locks and, as such, will better utilize your threads.

Boosting performance on async web calls

Backgound: I must call a web service call 1500 times which takes roughly 1.3 seconds to complete. (No control over this 3rd party API.) total Time = 1500 * 1.3 = 1950 seconds / 60 seconds = 32 minutes roughly.
I came up with what I though was a good solution however it did not pan out that great.
So I changed the calls to async web calls thinking this would dramatically help my results it did not.
Example Code:
Pre-Optimizations:
foreach (var elmKeyDataElementNamed in findResponse.Keys)
{
var getRequest = new ElementMasterGetRequest
{
Key = new elmFullKey
{
CmpCode = CodaServiceSettings.CompanyCode,
Code = elmKeyDataElementNamed.Code,
Level = filterLevel
}
};
ElementMasterGetResponse getResponse;
_elementMasterServiceClient.Get(new MasterOptions(), getRequest, out getResponse);
elementList.Add(new CodaElement { Element = getResponse.Element, SearchCode = filterCode });
}
With Optimizations:
var tasks = findResponse.Keys.Select(elmKeyDataElementNamed => new ElementMasterGetRequest
{
Key = new elmFullKey
{
CmpCode = CodaServiceSettings.CompanyCode,
Code = elmKeyDataElementNamed.Code,
Level = filterLevel
}
}).Select(getRequest => _elementMasterServiceClient.GetAsync(new MasterOptions(), getRequest)).ToList();
Task.WaitAll(tasks.ToArray());
elementList.AddRange(tasks.Select(p => new CodaElement
{
Element = p.Result.GetResponse.Element,
SearchCode = filterCode
}));
Smaller Sampling Example:
So to easily test I did a smaller sampling of 40 records this took 60 seconds with no optimizations with the optimizations it only took 50 seconds. I would have though it would have been closer to 30 or better.
I used wireshark to watch the transactions come through and realized the async way was not sending as fast I assumed it would have.
Async requests captured
Normal no optimization
You can see that the asnyc pushes a few very fast then drops off...
Also note that between requests 10 and 11 it took nearly 3 seconds.
Is the overhead for creating threads for the tasks that slow that it takes seconds?
Note: The tasks I am referring to are the 4.5 TAP task library.
Why wouldn't the request come faster than that.
I was told the Apache web server I was hitting could hold 200 max threads so I don't see an issue there..
Am I not thinking about this clearly?
When calling web services are there little advantages from async requests?
Do I have a code mistake?
Any ideas would be great.

After many days of searching I found this post that solved my problem:
Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)
The reason that the request was not hitting the server quicker was due too the my client side code and nothing to do with the server. By default C# only allows 2 concurrent requests.
see here: http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit.aspx
I simply added this line of code and then all request came through in milliseconds.
System.Net.ServicePointManager.DefaultConnectionLimit = 50;

How to verify if Parallel.Foreach is working correctly

Inside a Parallel Foreach I'm calling a service that gives me all information about an article, If I try to take the information about only one it takes 6 seconds.
My questions about that:
If I want to take the information about 4 articles, how long will it take? +- 6 seconds??
Actually doing that it takes 27 seconds, There's an easy way to check if it's working on parallel ??
Working with C# MVC3
Code:
private void PopulateArticleDictionary()
{
List<Article> tmpArticleFirstLevel = new List<Article>();
Parallel.ForEach<Article>(ArticlesFirstLevel,
article =>
{
var articleInDepth = ArticleService.SearchByCode(article.Code, article.Code18, article.Quantity, "ES", "EUR");
if (articleInDepth == null)
{
tmpArticleFirstLevel.Add(article);
}
else
{
tmpArticleFirstLevel.Add(articleInDepth);
}
}
);
ArticlesFirstLevel = tmpArticleFirstLevel;
}
Thank's !

How many cores do you have? The Parallel libraries won't spin up extra threads if you don't have the extra processors to take advantage of it. Also there is some overhead, doing parallel on 4 articles probably isn't really worth it unless you have a lot of crunching going on.
Also are you sure your service isn't a bottle neck?
Try this and see if there is an improvement
System.Net.ServicePointManager.DefaultConnectionLimit = 1000;
You could have a bottleneck connecting to your service as .NET out of the box only allows 2 concurrent HTTP pipes to the same host.