I'm trying to monitor a connection by performing regular 'pings' with a parallel 'timeout' task in case the ping doesn't come back in time.
var pingTask = await Task.WhenAny(
Task.Run(() =>
{
try
{
PingThing();
return true;
}
catch
{
return false;
}
}),
Task.Run(() =>
{
Task.Delay(5000).Wait();
return false;
})
);
if (!pingTask.Result) //etc...
I seem to be getting cases where 'other stuff' is taking priority, and this code gets put to one side. So where the pings normally take milliseconds, I'm getting five second timeouts.
The thing is, when I initially put a stopwatch inside the code to check, it didn't say anything like 5 seconds. Which has me wondering whether the CPU isn't shutting down my thread(s) to do other work. But that in turn makes me wonder why it would shut down the first Task, but keep the second one up (otherwise the 'timeout' Task would also take longer).
So I have several questions:
is this a plausible hypothesis?
is there a way to set the priority inside a task?
if so is that a good or bad idea?
is using await here a bad idea (does it somehow signal low priority)?
is there a way to monitor what all the threads are doing to try and work out what the 'other stuff' is that's stealing my processor time?
Leaky abstractions FTW...
Related
Always managing to confuse myself working with async, I'm after a bit of validation/confirmation here that i'm doing what i think i'm doing in the following scenarios..
given the following trivial example:
// pretend / assume these are json msgs or something ;)
var strEvents = new List<string> { "event1", "event2", "event3" };
i can post each event to an eventhub simply as follows:
foreach (var e in strEvents)
{
// Do some things
outEventHub.Add(e); // ICollector
}
the foreach will run on a single thread, and execute each thing inside sequentially.. the posting to eventhub will also remain on the same thread too i guess??
Changing ICollector to IAsyncCollector, and achieve the following:
foreach (var e in strEvents)
{
// Do some things
await outEventHub.AddAsync(e);
}
I think i am right here in saying that the foreach will run on a single thread, the actual sending to the event hub will be pushed off elsewhere? Or at least not block that same thread..
Changing to Parallel.ForEach event as these events will be arriving 100+ or so at a time:
Parallel.ForEach(events, async (e) =>
{
// Do some things
await outEventHub.AddAsync(e);
});
Starting to get a bit hazy now, as i am not sure what really is going on now... afaik the each event has it's own thread (within the bounds of the hardware) and steps within that thread do not block it.. so this trivial example aside.
Finally, i could turn them all in to Tasks i thought..
private static async Task DoThingAsync(string e, IAsyncCollector<string> outEventHub)
{
await outEventHub.AddAsync(e);
}
var t = new List<Task>();
foreach (var e in strEvents)
{
t.Add(DoThingAsync(e, outEventHub));
}
await Task.WhenAll(t);
now i am really hazy, and i think this is prepping everything on a single thread.. and then running everything exactly at the same time, on any thread available??
I appreciate that in order to determine which is right for the job at hand benchmarking is required... but an explanation of what the framework is doing in each situation would be super helpful for me right now..
Parallel != async
This is the main idea here. Both of them have their uses, and they can be used together, but they are very different. You are mostly right with your assumptions, but let me clarify:
Simple foreach
This is non-parallel and non-async. Nothing to talk about.
Await inside foreach
This is async code that is non-parallel.
foreach (var e in strEvents)
{
// Do some things
await outEventHub.AddAsync(e);
}
This will all take place on a single thread. It takes an event, starts adding it to your event hub, and while it is being completed (I'm guessing it does some sort of network IO) it hands back the thread to the thread pool (or UI if it was called on a UI thread) so it can do other work while wating on AddAsync to return. But as you said, is is not parallel at all.
Parallel Foreach (async)
This one is a trap! In short, Parallel.Foreach is designed for synchronous workloads. We'll get back to this but first let's assume you used it with the non-async code.
Parallel foreach (sync)
A.k.a. Parallel but not async.
Parallel.ForEach(events, (e) =>
{
// Do some things
outEventHub.Add(e);
});
Each item will get its own "Task", but they won't spawn a thread. Creating threads is expensive, and in an optimal case there is no point in having more threads than CPU cores. Instead these tasks run on a ThreadPool, which has just as many Threads as optimal. Each thread takes a task, works on it, then takes another one, etc.
You can think of it as - on a 4 core machine - having 4 workers around a pile of tasks, so 4 of them are being run at a time. You can imagine that this is not ideal in case of IO bound workloads (which this most likely is). If your network is slow, you can have all 4 threads blocked on trying to send the event out, while they could be doing useful work. This leads us to...
Tasks
Async and potentially parallel (depends on the usage).
Your description is correct here, too, except for the ThreadPool, it is kikking off all the tasks at once (on the main thread), which then run on the pool's threads. While they are running, the main thread is released, which then can do other work, as needed. Up to this point it is the same as the Parallel.Foreach case. But:
What happens is that a TaskPool thread picks up a task, does the necessary preprocessing, then sends out the network request asynchronously. This means that this task will not block while waiting for the network, but rather it releases the ThreadPool thread to pick up another workitem. When the network request completes, the tasks continuation (the remaining code lines after the network request) is scheduled back to the list of tasks.
You can see that theoretically this is the most efficient process, so fast that you have to be careful not to flood your network.
Back to Parallel.Foreach and async
At this point you should be able to spot the problem. All your async lambda async (e) => { await outEventHub.AddAsync(e);} is doing is to kick off the work, it will return right after it hits the await. (Remember that async/await is releasing threads while waiting.) Parallel.Foreach returns right after it started all of them. But nothing is awaiting these tasks! These become fire and forget, which is usually a bad practice. It is like you deleted the await Task.WhenAll call from your task example.
I hope this cleared most things for you, if not, let me know what to improve on.
Why don't you send those events asynchronously in parallel, like this:
var tasks = new List<Task>();
foreach( var e in strEvents )
{
tasks.Add(outEventHub.AddAsync(e));
}
await Task.WhenAll(tasks);
await outEventHub.FlushAsync();
A Windows Service uses too many threads. I added some logging to find out more. Sadly, there's little support from the .Net framework.
ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads); starts with some 32760 workerThreads, and 1000 completionThreads, resp.
After a few hours, available workerThreads went down to 31817, i.e. almost 1000 managed threads are in use.
What are they doing? There's no way to find out (you may find some workaround where you place the threads you create into some collection, and later analyze that collection, but that fails when you also use Parellel.ForEach or Task.Run).
Well, there is another possibility. Try ProcessThreadCollection currentThreads = Process.GetCurrentProcess().Threads; That will give you a list of non-managed threads (there number is also shown in Windows Task Manager).
My Windows Service starts with some 20 of them. After a few hours, I detect 3828, i.e. about 4 non-managed threads for each managed thread...
Now I can ask each of them when it started, what its priority is, what it is doing currently, and why it is waiting. Yes, for almost all of them the current state is Wait. And the WaitReason is in most cases UserRequest.
So my question is: what are those threads actually doing? There is no User Interface, even no command line associated with that executable: it is a Windows Service...
Also, I'd like to know how to get rid off them: many threads are created and should also run to completion in a short time (within seconds) - but some are "waiting" for hours.
I solved that issue by using the rationale that a thread which is not created cannot hang around uselessly.
I removed some calls to Parallel.Foreach(collection, item => { item.DoSomething(parameters); } );. Now implementations of IItem.DoSomething(parameters) just enqueue the parameters for later processing, and IItems have a thread for that processing (Active Object Pattern). Consequently, a "common" foreach can be used.
When results need to be collected, the pattern is more complicated:
private List<IResult> CollectResults(IEnumerable<IItem> collection, int maximumProcessingMilliseconds )
{
List<IResult> results = new List<IResult>();
CancellationTokenSource cts = new CancellationTokenSource();
cts.CancelAfter(maximumProcessingMilliseconds);
var tasks = new List<Task<IResult>>();
foreach (IItem item in collection)
{
IItem localItem = item;
tasks.Add(Task.Run(() => localItem.GetResult(cts.Token), cts.Token));
}
Task[] tasksArray = tasks.ToArray();
try
{
Task.WaitAll(tasksArray, TimeSpan.FromMilliseconds(maximumProcessingMilliseconds));
Task.WaitAll(tasksArray);
}
catch (AggregateException ex)
{
Logger.LogException(ex);
}
foreach (Task<IResult> task in tasks)
{
if (task.Status == TaskStatus.RanToCompletion)
{
results.Add(task.Result);
}
}
return results;
}
This is a terrible lot of boiler plate code which obfuscates the actual meaning of the function.
In the end, the number of (unmanaged) threads used by the application hardly ever grows beyond 100, and if it did, it does return to lower values quickly.
I'm working on a SMS-based game (Value Added Service), in which a question must be sent to each subscriber on a daily basis. There are over 500,000 subscribers and therefore performance is a key factor. Since each subscriber can be a difference state of the competition with different variables, database must be queried separately for each subscriber before sending a text message. To achieve the best performance I'm using .Net Task Parallel Library (TPL) to spawn parallel threadpool threads and do as much async operations as possible in each thread to finally send texts asap.
Before describing the actual problem there are some more information necessary to give about the code.
At first there was no async operation in the code. I just scheduled some 500,000 tasks with the default task scheduler into the Threadpool and each task would work through the routines, blocking on all EF (Entity Framework) queries and sequentially finishing its job. It was good, but not fast enough. Then I changed all EF queries to Async, the outcome was superb in speed but there has been so many deadlocks and timeouts in SQL server that about a third of the subscribers never received a text! After trying different solutions, I decided not to do too many Async Database operations while I have over 500,000 tasks running on a 24 core server (with at least 24 concurrent threadpool threads)!
I rolled back all the changes (the Asycn ones) expect for one web service call in each task which remained Async.
Now the weird case:
In my code, I have a boolean variable named "isCrossSellActive". When the variable is set some more DB operations take place and an asycn webservice call will happen on which the thread awaits. When this variable is false, none of these operations will happen including the async webservice call. Awkwardly when the variable is set the code runs so much faster than when it's not! It seems like for some reason the awaited async code (the cooperative thread) is making the code faster.
Here is the code:
public async Task AutoSendMessages(...)
{
//Get list of subscriptions plus some initialization
LimitedConcurrencyLevelTaskScheduler lcts = new LimitedConcurrencyLevelTaskScheduler(numberOfThreads);
TaskFactory taskFactory = new TaskFactory(lcts);
List<Task> tasks = new List<Task>();
//....
foreach (var sub in subscriptions)
{
AutoSendData data = new AutoSendData
{
ServiceId = serviceId,
MSISDN = sub.subscriber,
IsCrossSellActive = bolCrossSellHeader
};
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
}
GC.Collect();
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException ae)
{
ae.Handle((ex) =>
{
_logRepo.LogException(1, "", ex);
return true;
});
}
await _autoSendRepo.SetAutoSendingStatusEnd(statusId);
}
public async Task SendQuestion(object data)
{
//extract variables from input parameter
try
{
if (isCrossSellActive)
{
int pieceCount = subscriptionRepo.GetSubscriberCarPieces(curSubscription.service, curSubscription.subscriber).Count(c => c.isConfirmed);
foreach (var rule in csRules)
{
if (rule.Applies)
{
if (await HttpClientHelper.GetJsonAsync<bool>(url, rule.TargetServiceBaseAddress))
{
int noOfAddedPieces = SomeCalculations();
if (noOfAddedPieces > 0)
{
crossSellRepo.SetPromissedPieces(curSubscription.subscriber, curSubscription.service,
rule.TargetShortCode, noOfAddedPieces, 0, rule.ExpirationLimitDays);
}
}
}
}
}
// The rest of the code. (Some db CRUD)
await SmsClient.SendSoapMessage(subscriber, smsBody);
}
catch (Exception ex){//...}
}
Ok, thanks to #usr and the clue he gave me, the problem is finally solved!
His comment drew my attention to the awaited taskFactory.StartNew(...) line which sequentially adds new tasks to the "tasks" list which is then awaited on by Task.WaitAll(tasks);
At first I removed the await keyword before the taskFactory.StartNew() and it led the code towards a horrible state of malfunction! I then returned the await keyword to before taskFactory.StartNew() and debugged the code using breakpoints and amazingly saw that the threads are ran one after another and sequentially before the first thread reaches the first await inside the "SendQuestion" routine. When the "isCrossSellActive" flag was set despite the more jobs a thread should do the first await keyword is reached earlier thus enabling the next scheduled task to run. But when its not set the only await keyword is the last line of the routine so its most likely to run sequentially to the end.
usr's suggestion to remove the await keyword in the for loop seemed to be correct but the problem was the Task.WaitAll() line would wait on the wrong list of Task<Task<void>> instead of Task<void>. I finally used Task.Run instead of TaskFactory.StartNew and everything changed. Now the service is working well. The final code inside the for loop is:
tasks.Add(Task.Run(async () =>
{
await SendQuestion(data);
}));
and the problem was solved.
Thank you all.
P.S. Read this article on Task.Run and why TaskFactory.StartNew is dangerous: http://blog.stephencleary.com/2013/08/startnew-is-dangerous.html
It's extremly hard to tell unless you add some profiling that tell you which code is taking longer now.
Without seeing more numbers my best guess would be that the SMS service doesn't like when you send too many requests in a short time and chokes. When you add the extra DB calls the extra delay make the sms service work better.
A few other small details:
await Task.WhenAll is usually a bit better than Task.WaitAll. WaitAll means the thread will sit around waiting. Making a deadlock slightly more likely.
Instead of:
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
You should be able to do
tasks.Add(SendQuestion(data));
I have a "worker" process that is running constantly on a dedicated server, sending emails, processing data extracts etc.
I want to have all of these processes running asynchronously, but I only want one instance of each process running at any one time. If a process is already running, I don't want to queue up running it again.
[example, simplified]
while (true)
{
// SLEEP HERE
Task task1 = Task.Factory.StartNew(() => DataScheduleWorker.Run());
Task task2 = Task.Factory.StartNew(() => EmailQueueWorker.Run());
}
Basically, I want this entire process to run endlessly, with each of the tasks running parallel to each other, but only one instance of each task running at any point in time.
How can I achieve this in C# 5? What's the cleanest/best way to implement this?
EDIT
Would something as simple as this suffice, or would this be deemed bad?:
Task dataScheduleTask = null;
while (true)
{
Thread.Sleep(600);
// Data schedule worker
if (dataScheduleTask != null && dataScheduleTask.IsCompleted) dataScheduleTask = null;
if (dataScheduleTask == null)
{
dataScheduleTask = Task.Factory.StartNew(() => DataScheduleWorker.Run());
}
}
This sounds like a perfect job for either an actors framework, or possibly TPL Dataflow. Fundamentally you've got one actor (or block) for each job, waiting for messages and processing them independently of the other actors. In either case, your goal should be to write as little of the thread handling and message passing code as possible - ideally none. This problem has already been largely solved; you just need to evaluate the available options and use the best one for your task. I would probably start with Dataflow, personally.
I have created a timeout function based on things I have seen in various places but am pretty sure I am not doing it a great way! (But it does seem to work.)
I am connecting to a piece of hardware that if working connects in a few seconds but if not takes around 1 minute to timeout. So if I can create my own timeout function I can set it at 20 seconds and save lots of time and waiting.
I have tried to make it so my timeout returns a string:
static string CallWithTimeout(Action action, int timeoutMilliseconds)
{
string reply = "";
Thread threadToKill = null;
Action wrappedAction = () =>
{
threadToKill = Thread.CurrentThread;
action();
};
IAsyncResult result = wrappedAction.BeginInvoke(null, null);
if (result.AsyncWaitHandle.WaitOne(timeoutMilliseconds))
{
reply = "Connected";
wrappedAction.EndInvoke(result);
return reply;
}
else
{
threadToKill.Abort();
reply = "Error";
return reply;
}
}
then I call it with something like :
string replyfromreader = CallWithTimeout(connectToHardware, 20000);
the connectToHardware is just a one liner so no need to post.
It's okayish as far as .NET state is concerned. You won't call EndInvoke(), that leaks resources for 10 minutes, the default lifetime of remoted objects.
In a case like this, calling Thread.Abort() has a very small chance of succeeding. A managed thread needs to be in an alertable wait state to be abortable, it just never is when the thread is buried deep inside native code that ultimately waits for some device driver call to complete.
Leaving the CLR in a state where it keeps trying to abort a thread and never succeeds is not particularly pleasant, not something I've ever tried on purpose so no real idea what the side-effects are. It does however mean that your code will block on the Abort() method call so you still haven't fixed the problem. The best thing to do is therefore to not abort the thread but just abandon it. Setting a flag that marks the device dead so you don't try to do this ever again.
If you want to continue running your program, even without the device being in a usable state, and you want to provide a way to recover from the problem then you'll need an entirely different approach. You'll need to put the device related code in a separate process. Which you can then Kill() when the device is unresponsive, relying on Windows to clean up the shrapnel. Interop with that process using a low-level mechanism like named pipes or sockets is best so you can recover from the disconnect fairly easily.
Avoiding Thread.Abort is always a good idea. Avoiding it on a thread you did not create is even better.
Assuming if the hardware is not working, and you want the timeout, it does not matter if connectToHardware is left to timeout on its own and no error/exception details are wanted, then you can use the Task Parallel Library (TPL): System.Threading.Tasks.Task:
// True => worked, False => timeout
public static bool CallWithTimeout(Action method, TimeSpan timeout) {
Exception e;
Task worker = Task.Factory.StartNew(method)
.ContineueWith(t => {
// Ensure any exception is observed, is no-op if no exception.
// Using closure to help avoid this being optimised out.
e = t.Exception;
});
return worker.Wait(timeout);
}
(If the passed Action could interact with a passed CancellationToken this could be made cleaner, allowing the underlying method to fail quickly on timeout.)