I'm trying to determine if something exists that allows you to run code in a distributed way. Each code task will have an Id and allow for smart scheduling.
Does not allow code with the same ID to run at the same time. Task will wait in a queue before being executed.
Task can trigger other tasks to be run.
If a multiple tasks trigger the same task (determined by id) to run, only one task will be triggered and not two.
It shouldn't be too difficult to write your own task scheduler and deploy it as a Windows service. This way you can ensure the scheduling rules you have outlined are followed.
Otherwise, you can have a look at Quartz.NET and see if it meets your needs.
Related
I have a website page that needs the option of performing an operation that could take several minutes. To avoid performance issues and time outs, I want to run this operation outside of the HTTP request.
After some research, I found IHostedService and BackgroundService, which can be registered as a singleton using AddHostedService<T>().
But my concern is that a hosted service is always running. Doesn't that seem like a waste of resources when I just want it to run on demand?
Does anyone know a better option to run a lengthy task, or a way to use IHostedService that doesn't need to run endlessly?
Note that the operation calls and waits for an API call. And so I cannot report the progress of the operation, nor can I set a flag in a common database regarding whether the operation has completed.
One option to run a lengthy task on demand while avoiding performance issues and time outs is to use a message queue. You can have your Razor Pages website send a message to the queue when the operation is requested, and have a separate service, such as a background worker, consume messages from the queue and perform the operation. This allows you to decouple the task from the web request, and also allows for the possibility of adding more worker instances to handle the workload.
Another option is to use a task scheduler that runs on demand, such as Hangfire. It allows you to schedule background jobs and monitor their progress, which can be useful in your scenario where you cannot report the progress of the operation.
You can also use IHostedService, but you need to make sure that the service is only running when it is needed. You can use a flag or a semaphore to control whether the service is running or not. You can set the flag or semaphore when the operation is requested, and clear it when the operation is completed. The service can then check the flag or semaphore in its main loop, and exit if the flag is not set.
In summary:
message queue, task scheduler, and IHostedService with controlling flag/semaphore are all viable options for running a lengthy task on demand. The best option depends on your specific use case and requirements.
I have two situations that I'm not sure how to resolve properly. First is a task that runs for as long as the application. Currently, I have a construction like this:
this.task = this.joinableTaskFactory.RunAsync(async () => await this.TaskProcedureAsync().ConfigureAwait(false), JoinableTaskCreationOptions.LongRunning);
Here TaskProcedureAsync is usually a loop until the application end signal is received and inside of it is await of some sort that provides a job to complete (e.g. from an async queue), when it arrives, it is processed and then it waits again for the next job.
Here I'm not sure if running with JoinableTaskCreationOptions.LongRunning is a correct solution for such a task. In DisposeCoreAsync then I await this joinable task:
await this.task.JoinAsync().ConfigureAwait(false);
and then do the rest of the disposal work. So the question here, if this is right thing to do in such scenario or if there is a better way to run such long running async task. This seems to work fine as it is, just not sure if it is optimal regarding the internal management of threads. There are not that many of these loops in the application, up to 30 is a reasonable expectation for my application.
The follow up problem I am quite sure that does not work well as I have it now. Besides those loop tasks that I have that run for as long as the application, I also have a bigger number of long running task that are supposed to be strictly time limited. Currently I use TplExtensions.WithTimeout extension:
result = await script.InitializeAsync().WithTimeout(TimeSpan.FromSeconds(ScriptInitializationTimeoutSeconds)).ConfigureAwait(false);
Basically I am running an untrusted code that must be stopped after certain limit. Now the problem is that WithTimeout does not stop the task itself, only throws exception after the time is out. The question here is how to implement this properly? I was thinking that if the code is to be stopped, it should be run in its own thread as a thread can be killed while task can not. But then I've read here that such an approach would not work because I could only execute synchronous code in such a thread and not async code.
Is there any way to kill the task then? Can I for example kill the thread that currently executes the task? Or how is this done properly?
I have an async call (DoAsyncWork()), that I would like to start in a fire-and-forget way, i.e. I'm not interesting in its result and would like the calling thread to continue even before the async method is finished.
What is the proper way to do this? I need this in both, .NET Framework 4.6 as well as .NET Core 2, in case there are differences.
public async Task<MyResult> DoWorkAsync(){...}
public void StarterA(){
Task.Run(() => DoWorkAsync());
}
public void StarterB(){
Task.Run(async () => await DoWorkAsync());
}
Is it one of those two or something different/better?
//edit: Ideally without any extra libraries.
What is the proper way to do this?
First, you need to decide whether you really want fire-and-forget. In my experience, about 90% of people who ask for this actually don't want fire-and-forget; they want a background processing service.
Specifically, fire-and-forget means:
You don't care when the action completes.
You don't care if there are any exceptions when executing the action.
You don't care if the action completes at all.
So the real-world use cases for fire-and-forget are astoundingly small. An action like updating a server-side cache would be OK. Sending emails, generating documents, or anything business related is not OK, because you would (1) want the action to be completed, and (2) get notified if the action had an error.
The vast majority of the time, people don't want fire-and-forget at all; they want a background processing service. The proper way to build one of those is to add a reliable queue (e.g., Azure Queue / Amazon SQS, or even a database), and have an independent background process (e.g., Azure Function / Amazon Lambda / .NET Core BackgroundService / Win32 service) processing that queue. This is essentially what Hangfire provides (using a database for a queue, and running the background process in-proc in the ASP.NET process).
Is it one of those two or something different/better?
In the general case, there's a number of small behavior differences when eliding async and await. It's not something you would want to do "by default".
However, in this specific case - where the async lambda is only calling a single method - eliding async and await is fine.
It depends on what you mean by proper :)
For instance: are you interested in the exceptions being thrown in your "fire and forget" calls? If not, than this is sort of fine. Though what you might need to think about is in what environment the task lives.
For instance, if this is a asp.net application and you do this inside the lifetime of a thread instantiated due to a call to a .aspx or .svc. The Task becomes a background thread of that (foreground)thread. The foreground thread might get cleaned up by the application pool before your "fire and forget" task is completed.
So also think about in which thread your tasks live.
I think this article gives you some useful information on that:
https://www.hanselman.com/blog/HowToRunBackgroundTasksInASPNET.aspx
Also note that if you do not return a value in your Tasks, a task will not return exception info. Source for that is the ref book for microsoft exam 70-483
There is probably a free version of that online somewhere ;P https://www.amazon.com/Exam-Ref-70-483-Programming-C/dp/0735676828
Maybe useful to know is that if your have an async method being called by a non-async and you wish to know its result. You can use .GetAwaiter().GetResult().
Also I think it is important to note the difference between async and multi-threading.
Async is only useful if there are operations that use other parts of a computer that is not the CPU. So things like networking or I/O operations. Using async then tells the system to go ahead and use CPU power somewhere else instead of "blocking" that thread in the CPU for just waiting for a response.
multi-threading is the allocation of operations on different threads in a CPU (for instance, creating a task which creates a background thread of the foreground thread... foreground threads being the threads that make up your application, they are primary, background threads exist linked to foreground threads. If you close the linked foreground thread, the background thread closes as well)
This allows the CPU to work on different tasks at the same time.
Combining these two makes sure the CPU does not get blocked up on just 4 threads if it is a 4 thread CPU. But can open more while it waits for async tasks that are waiting for I/O operations.
I hope this gives your the information needed to do, what ever it is you are doing :)
I will use "Process" to refer to the work that is going to happen in parallel, and "enqueue" to refer to whatever process is going to be used to initiate that process (whether that be Task.Run, ThreadPool.QUWI, new Thread() ... whatever).
We have a performance sensitive program that spawn multiple parallel processes to gather data.
We're having issues with the spawning, that the processes are not beginning immediately.
Specifically, if we prepare a process, start a timer, enqueue the process, and check the timer as the very first action in the process ... then we see that the time delay occasionally stretches into 100s or even 1000s of milliseconds.
Given that the process itself is supposed to only run for 3-10 seconds, having a 2sec delay between enqueuing and activation of the process is a major issue.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Currently our implementations started using TP.QUWI, and then we move to using Task.Run.
Our initial investigation lead us to the Thread-Creation-Strategy used by Threadpool and using ThreadPool.SetMinThreads(), so we're pursuing that angle, to see if that will completely resolve the issue.
But is there another change/approach that we should be looking at, if our goal is to have the process start immediately after enqueuing?
Taken from here (I strongly suggest you have a read up):
Seems as though what you want can be achieved by overridding the default task scheduler.... scarey...
You can't assume that all parallel tasks will immediately run. Depending on the current work load and system configuration, tasks might be scheduled to run one after another, or they might run at the same time. For more information about how tasks are scheduled, see the section, "The Default Task Scheduler," later in this chapter.
Creating Tasks with Custom Scheduling
You can customize the details of how tasks in .NET are scheduled and run by overriding the default task scheduler that's used by the task factory methods. For example, you can provide a custom task scheduler as an argument to one of the overloaded versions of the TaskFactory.StartNew method.
There are some cases where you might want to override the default scheduler. The most common case occurs when you want your task to run in a particular thread context... Other cases occur when the load-balancing heuristics of the default task scheduler don't work well for your application. For more information, see the section, "Thread Injection," later in this chapter.
Unless you specify otherwise, any new tasks will use the current task scheduler...
You can implement your own task scheduler class. For more information, see the section, "Writing a Custom Task Scheduler," later in this chapter.
Thread Injection
The .NET thread pool automatically manages the number of worker threads in the pool...
Have a read of this SO post "replacing the task scheduler in c sharp with a custom built one"
My issue is pretty simple.
I have an application that should be executed automatically once a day. I have no prior experience with this kind of scenario (some time ago I worked with IBM Control-M but guess that it is way more complete, complex and expensive =))
I thought about two possible solutions:
Creating a Task inside Windows Task Scheduler, which would execute the application;
Implement the application as a Window Service which would run 24/7, but only would perform the needed actions depending on the current time.
Which are the advantages/disadvantages of each approach?
Is there another way to do this?
Thanks in advance.
If it only executes once a day (or so) then just do it as a regular command line app that is executed by the windows task scheduler. Task scheduler already has all of the UI necessary to determine when to kick off the program, pass in parameters and anything else related to the scheduling of the task.
The only real reason to do this type of function as a windows service is if it needs higher execution resolution than once a minute. However, the main downside to a windows service is that you would have to manage the logic for how often/when to kick it off. Another one is that the app is always running, which leaves open the possibility for leaked memory if your code has issues.
On Unix/Linux you would use a cron job schedule a task to be executed. MS Windows' version is called the Task Scheduler and it is already a service that run 24/7 and performs the needed actions depending on the time.
Create a repeating task with the Task Scheduler to run your application. Creating, installing and configuring a service application is not exactly trivial. It's a much more involved process than creating a standard Forms or command line app and you don't need to do it anyway.
http://msdn.microsoft.com/en-us/magazine/cc164015.aspx
http://www.dotnetmonster.com/Uwe/Forum.aspx/dotnet-csharp/70633/Waitable-Timer-in-C
Another library that might be of interest is Quartz.NET