Sorry if this is a duplicate, but I'm not quite sure what terms I need to use to find existing answers to this question.
I'm trying to improve start-up performance of an application, the pseudo-code looks a bit like this.
LoadBigFileFromDisk(); //slow
SetupNetwork(); //even slower
UseBigFileFromDisk();
I figured that as the first step is disk-bound, and the other network-bound (and slower), I could run the first in a background thread (currently playing with ThreadPool.QueueUserWorkItem, but not sure if that's the best way) and improve the performance a bit.
It works, but what worries me is that I'm relying on the second step being slow enough for the first to complete.
I know I could set a _done boolean somewhere and while ! on that, but is there a more elegant/idiomatic solution?
(Not .Net 4.0 yet, so though I'm interested in Task-based, I need the fall-back solutions).
In the "main class" do this:
ManualResetEvent mre = new ManualResetEvent(false);
in your "main" method do this:
// Launch the tasks
mre.WaitOne();
in the task when it finishes (just before the return :-) )
mre.Set();
If you have to wait for multiple events, in your "main" create multiple ManualResetEvent and put them in an array, each event "connected" to one of the tasks, and then each task Sets its event when it finishes. Then in your "main" you do:
WaitHandle.WaitAll(arrayOfManualResetEvents);
Note that in this way you can wait up to 64 events. If you need more, there is another method (and note that you have to use this last method even if you are on a STA thread, like the main thread of a WinForm app).
ManualResetEvent mre = new ManualResetEvent(false);
int remaining = xxx; // Number of "tasks" that must be executed.
// Launch tasks
mre.WaitOne();
At the end of each task
if (Interlocked.Decrement(ref remaining) == 0)
{
mre.Set();
}
Only the last task will decrement the remaining field to 0 and mre.Set().
You could try:
var loadTask=Task.Factory.StartNew(()=>LoadBigFileFromDisk());
var setupTask=Task.Factory.StartNew(()=>SetupNetwork());
Task.WaitAll(loadTask,setupTask);
UseBigFileFromDisk();
This uses the Task Parallel Library.
or:
var loadThread=new Thread(()=>LoadBigFileFromDisk());
var setupThread=new Thread(()=>SetupNetwork());
loadThread.Start();
setupThread.Start();
loadThread.Join();
setupThread.Join();
UseBigFileFromDisk();
When you're not using .NET 4. If these tasks take a long time to run then it's best to avoid the thread pool, as it's primarily for short lived tasks.
Try Thread.Join . Something like networkThread.Join()
http://msdn.microsoft.com/en-us/library/95hbf2ta.aspx
Related
I using python 3.6 for sync multiple threads. I have a "master thread" that gives work for all the other threads. When a worker thread is finish work, it signal the master thread to give him more work.
In order to achive that, the master thread is waiting for one (or more) threads to finish before collecting new data to process.
while True:
while freeWorkers > 0:
# Give the worker more work...
time.sleep(5) # wait for 5 seconds before checking if we got free workers.
Basiclly, it's working. I want to upgrade it in that way: after a worker finish it job, it report some how to the "master" thread. Because master thread is really quick, in most cases the master thread will be sleeping... I want to make him stop sleeping, what will trigger giving more work for the free workers.
In C#, I did this trick in that way:
An object to handle the syncing around
public object SyncingClock { get; private set; } = new object();
Entering sleep in that way:
lock (SyncingClock)
Monitor.Wait(SyncingClock, 5000);
Worker thread report completion in that way:
lock (SyncingClock)
Monitor.Pulse(SyncingClock);
So, I looking to way to perform this C# trick in Python (or any other alternative).
Thanks.
i think you should look at eventdriven programming (https://emptypage.jp/notes/pyevent.en.html)
and not having a while loop polling for finished workers:
for example something like this:
def create_thread(self, work_finished_method):
t = some_method_to_create_and prepare_a_thread()
t.event_finished += work_finished_method
return t
class MyThread:
name = "SomeNameForTheThread"
event_finished = event.Event(name + " has finished.")
def finished(self):
self.event_finished()
def do_work:
do_something()
finished()
and when the work_finished method is called in the mainhthread you can assign new work to the thread.
This done with Condition object.
self.conditon = threading.Condition()
For waiting to timeout or pulse, do:
with service.conditon:
service.conditon.wait(5)
For notify:
with self.conditon:
self.conditon.notifyAll()
I made a short program which has just a button. When the button is pressed, functionA is executed, which also uses functionB and functionC. Inside functionA is a loop which executes functionB and functionC X amount of times. At the end of each loop, the progressbar gets incremented by 1. At the beginning of functionA, before the loop, there's a webservice which pulls data from a website, and passes that onto B and C for processing (data file manipulation and saving to disk).
My problem is that everything works fine, but while functionA is still running, the GUI is stuck, so I can't close/minimize/drag the window around, I have to wait until A is done. I researched and they say I should use BackgroundWorker, but as being a new programmer, I've no idea on how to use it. Can someone give me a simple way to use it?
The progressbar loads fine, but it's just that while the function is running, the whole window is frozen, and I want it so I can move the window around, etc while the program is running, instead of waiting until the function is complete.
Thank you!
Call your function asynchronously like the following and it will not freeze the UI.
private async void BeginProcessingAsync(Data d)
{
//Execute the long running task asynchronously
await Task.Run(() => functionA(d));
//Anything after the await line will be executed only after the task is finished.
anotherFunction(d); // if you have one..
}
To run your task, simply call BeginProcessingAsync(d);. Also, please note: If you're using newer versions of .NET, you might have to use await Task.Factory.StartNew(() => functionA(d)); instead of the above
Overall, you'll want to make sure your GUI doesn't get updated from another thread. Instead, the messages should go to a threadsafe location. For instance, you could have the thread building into something like a database and have the GUI using a timer to look for updated data flags.
There is a question with a lot more detail using delegates here.
Marc's answer was the simplest and best, in my opinion:
///...blah blah updating files
string newText = "abc"; // running on worker thread
this.Invoke((MethodInvoker)delegate {
someLabel.Text = newText; // runs on UI thread
});
///...blah blah more updating files
From Dotnet Perls:
A Background Worker makes threads easy to implement in Windows
Forms. Intensive tasks need to be done on another thread so the UI
does not freeze. It is necessary to post messages and update the user
interface when the task is done.
Also, from MSDN, look at Task-based Asynchronous Pattern (TAP) if you're using C# 5.
The Task-based Asynchronous Pattern (TAP) is based on the
System.Threading.Tasks.Task and System.Threading.Tasks.Task
types in the System.Threading.Tasks namespace, which are used to
represent arbitrary asynchronous operations. TAP is the recommended
asynchronous design pattern for new development.
I am trying to use tasks in a little .net 4.0 application (written using Visual Studio 2010 if that matters) that needs to work on Windows 2003 and use a WriteableBitmap with the palette parameter.
The code using said class must, therefore, be running as an STA thread to avoid it throwing an invalid cast exception (see here for why I need an STA thread if you are interested, but it is not the thrust of my question).
I, therefore, checked on Stack overflow and came across How to create a task (TPL) running a STA thread? and The current SynchronizationContext may not be used as a TaskScheduler - perfect, so now I know what to do, except...
Here's a little console application:
using System;
using System.Threading;
using System.Threading.Tasks;
namespace TaskPlayingConsoleApplication
{
class Program
{
[STAThread]
static void Main()
{
Console.WriteLine("Before Anything: "
+ Thread.CurrentThread.GetApartmentState());
SynchronizationContext.SetSynchronizationContext(
new SynchronizationContext());
var cts = new CancellationTokenSource();
var scheduler = TaskScheduler.FromCurrentSynchronizationContext();
var task = Task.Factory.StartNew(
() => Console.WriteLine(
"In task: " + Thread.CurrentThread.GetApartmentState()),
cts.Token,
TaskCreationOptions.None,
scheduler);
task.ContinueWith(t =>
Console.WriteLine(
"In continue: " + Thread.CurrentThread.GetApartmentState()),
scheduler);
task.Wait();
}
}
}
And here is its output:
Before Anything: STA
In task: STA
In continue: MTA
What the!?! Yup, it is back to an MTA thread on the Action<Task> passed into the ContinueWith method.
I am passing the same scheduler into the task and the continue but somehow in the continue it seems to be being ignored.
I'm sure it is something stupid, so how would I make sure that my callback passed into the ContinueWith uses an STA thread?
EDIT: before you read any of the following, here's an excellent on-topic article: http://blogs.msdn.com/b/pfxteam/archive/2012/01/20/10259049.aspx ; You can skip my post and go directly there!
Most important part describing the root cause:
The default implementation of SynchronizationContext.Post just turns around and passes it off to the ThreadPool via QueueUserWorkItem. But (...) can derive their own context from SynchronizationContext and override the Post method to be more appropriate to the scheduler being represented.
In the case of Windows Forms, for example, the WindowsFormsSynchronizationContext implements Post to pass the delegate off to Control.BeginInvoke. For DispatcherSynchronizationContext in WPF, it calls to Dispatcher.BeginInvoke. And so on.
So, you need to use something other than the base SynchronizationContext class. Try using any of the other existing ones, or create your own. Example is included in the article.
And now, my original response:
After thinking a bit, I think the problem is that in your console application there is no thing like "message pump". The default SynchronizationContext is just a piece of lock. It prevents threads from intersecting on a resource, but it does not provide any queueing or thread selection. In general you are meant to subclass the SynchroContext to provide your own way of proper synchronization. Both WPF and WinForms provide their own subtypes.
When you Wait on your task, most probably the MainThread gets blocked and all other are run on some random threads from the default threadpool.
Please try writing Thread IDs to the console along with the STA/MTA flag.
You will probably see:
STA: 1111
STA: 1111
MTA: 1234
If you see this, then most probably your first task is run synchronously on the calling thread and gets instantly finished, then you try to "continue" it's just 'appended' to 'the queue', but it is not started immediatelly (guessing, I dont know why so; the old task is finished, so ContinueWith could also just run it synchronously). Then main thread gets locked on wait, and since there's no message pump - it cannot switch to another job and sleeps. Then threadpool waits and sweps the lingering continuation task. Just guessing though. You could try to check this by
prepare synccontext
write "starting task1"
start task1 ( -> write "task1")
write "continuing task2" <--- add this one
continue: task2 ( -> write "task2")
wait
and check the order of messages in the log. Is "continuing" before "hello" from task1 or not?
You may also try seeing what happens if you don't create the Task1 by StartNew, but rather create it as prepared/suspended, then Continue, then start, then wait. If I'm right about the synchronous run, then in such setup main and continuation task will either both be run on the calling '1111' STA thread, or both on threadpool's '2222' thread.
Again, if all of these is right, the providing some message pump and proper SyncContext type will probably solve your issue. As I said, both WPF and WinForms provide their own subtypes. Although I don't remember the names now, you can try using them. If I remember correctly, the WPF starts its dispatcher automatically and you don't need any extra setup. I don't remember how's with WinForms. But, with the WPF's auto-start, if your ConsoleApp is actually some kind of a unit-test that will run many separate cases, you will need to shutdown the WPF's dispatcher before the cases.. but that's far from the topic now.
I have a little c# app with multiple threads runing, but my main thread has to wait for all of threads to finish then it can do the rest.
problem now is that im using .join() for each thread, this seems wait for each thread to finish then it goes to next thread, which makes app not really multi-threading and take long time to finish.
so I wonder if there is any way I can get around this problem or just a way to check if there are no more threads is active.
thanks
If you're hanging on to the Thread object, you can use Thread.IsAlive.
Alternately, you might want to consider firing an event from your thread when it is done.
Thread.Join() doesn't mean your application isn't multithreaded - it tells the current thread to wait for the other thread to finish, which is exactly what you want.
Doing the following:
List<Thread> threads = new List<Thread>();
/** create each thread, Start() it, and add it to the list **/
foreach (Thread thread in threads)
{
thread.Join()
}
will continue to run the other threads, except the current/main thread (it will wait until the other threads are done).
Just use Thread.Join()
Ye, as said by Cuong Le, using Task Parallel Library would be much efficient.
However, you can Create a list of Threads and then check if they are alive or not.
var threadsList = new List<Thread>();
threadsList.Add(myThread); // to add
bool areDone = true;
foreach (Thread t in threadsList) {
if (t.IsAlive)
{
areDone = false;
break;
}
}
if (areDone)
{
// Everything is finished :O
}
Run multiple at same time but wanted to wait for all of them to finish, here's a way of doing the same with Parallel.ForEach:
var arrStr = new string[] {"1st", "2nd", "3rd"};
Parallel.ForEach<string>(arrStr, str =>
{
DoSomething(str); // your custom method you wanted to use
Debug.Print("Finished task for: " + str);
});
Debug.Print("All tasks finished");
That was the most simplest and efficient i guess it can go if in C# 4.0 if you want all tasks to run through same method
Try using BackgroundWorker
It raises an event in the main thread (RunWorkerCompleted) after its work is done
Here is one sample from previously answered question
https://stackoverflow.com/a/5551376/148697
I've got a WPF application that does a lot of talking to a remote server, so to keep the UI nice and responsive I put those operations in a second thread. There are a few possible, though unlikely, instances where that thread would just hang, blocking forever. Is there a simple way for me to implement a "cancel" button that doesn't involve calling thread.Abort()? I see a lot of people advise against using that, and I don't want to leave any unreleased resources. Perhaps a way to force the thread to throw an exception?
I specify in the title that this isn't a background worker because the program doesn't use those. It's already coded up with plain old threads.
I absolutely agree with comments that you need to fix what's broken, because that's the real issue, but if you can't at the moment, and need to continue operating. If you are on .Net 4.0, use the Task library. You can create a task and pass it an action that will execute on a different thread. The key is that you can pass a cancellation token to that task.
http://msdn.microsoft.com/en-us/library/dd997396.aspx
if you fire the same action over and over, you can also check the task status and do something about that.
//this is just a sample
if (myTask != null) //meaning it's still exe'ing your action
{
if (myTask.Status == TaskStatus.Faulted) //there's some kind of a problem with exe'ing it
myTask = null; // could reset to run the action again
else
return; //let the task finish
}
myTask = Task.Factory.StartNew (() =>
{
ExecuteUMyAction ();
});