I am planning to use quartz.net for processing files.
I will receive 1 or more files every hour and each file will have 1000’s of rows for different countries.
I want to read those rows, validate, process and insert it into multiple tables.
I want to pick 1 file and for each distinct country in that file I like to create a thread. Such that 1 thread will handle all data for 1 country.
At a given time there should not be more than 5 threads.
Now how do I define this in quartz.net? The below is the code I have in which I am going through each file by file and each row by row and I am not doing any multithreading
Scheduling a job
var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "MyScheduler";
properties["quartz.threadPool.threadCount"] = "5";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
IScheduler s = sf.GetScheduler();
if (!s.IsStarted)
s.Start();
var jobKey = new JobKey("UniqueJobName", "BatchProcess");
if (s.GetJobDetail(jobKey) != null)
return "Error! Already running";
IJobDetail jobDetail = JobBuilder.Create<CountryProcessJob>()
.WithIdentity(jobKey)
.Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("UniqueTriggerName", "BatchProcess")
.StartAt(DateTime.Now.AddSeconds(1))
.Build();
s.ScheduleJob(jobDetail, trigger);
Job
public class CountryProcessJob : IJob
{
public void Execute(IJobExecutionContext context)
{
While() // TillAllFilesAreProcessed
{
// Read A File
While() //for each row in file
{
// validate
// Process
// Insert
}
}
}
}
Should I have a main Job to loop through the Files one at time and then with in the Job should I define multiple Jobs for processing each country? Is this how to achieve multi threading in quartz?
Schedule the main Job
var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "MyScheduler";
properties["quartz.threadPool.threadCount"] = "5";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
IScheduler s = sf.GetScheduler();
if (!s.IsStarted)
s.Start();
var jobKey = new JobKey("UniqueJobName", "BatchProcess");
if (s.GetJobDetail(jobKey) != null)
return "Error! Already running";
IJobDetail jobDetail = JobBuilder.Create<FileProcessJob>()
.WithIdentity(jobKey)
.Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("UniqueTriggerName", "BatchProcess")
.StartAt(DateTime.Now.AddSeconds(1))
.Build();
s.ScheduleJob(jobDetail, trigger);
Main Job
public class FileProcessJob : IJob
{
public void Execute(IJobExecutionContext context)
{
while() // TillAllFilesAreProcessed
{
// Read A File
foreach(var eachCountry in file) //for each row in file
{
// Create a Job
var jobKey = new JobKey("UniqueCountryJobName", "BatchProcess");
if (s.GetJobDetail(jobKey) != null)
return "Error! Already running";
IJobDetail jobDetail = JobBuilder.Create<CountryProcessJob>()
.WithIdentity(jobKey)
.Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("UniqueCountryTriggerName", "BatchProcess")
.StartAt(DateTime.Now.AddSeconds(1))
.Build();
s.ScheduleJob(jobDetail, trigger);
}
}
}
}
Create multiple Jobs for each country
public class CountryProcessJob : IJob
{
public void Execute(IJobExecutionContext context)
{
// For each row for that country
// validate
// Process
// Insert
}
}
So should I create multiple Jobs to implement multiple thread to run at same time? Please help me to run 5 concurrent threads processing each distinct country with in a file.
Quartz.Net Jobs run in a seperate thread. so if you want to configure the max nuber of threads take a look at this answer: https://stackoverflow.com/a/4108795/745011
You can create a separate static class, and call it in the execute method. An static class have a fixed space in the memory.
public class ReadCountry : IJob
{
public void Execute(IJobExecutionContext context)
{
CountryProcessJob.DoIt();
}
}
CountryProcessJob class
public static class CountryProcessJob
{
public static void DoIt()
{
While() // TillAllFilesAreProcessed
{
// Read A File
While() //for each row in file
{
// validate
// Process
// Insert
}
}
}
}
Related
I Just Want To Trigger Event Only once in a Day At specific Time But it trigger 4 Times with difference of some miliseconds
Below Is My Scheduler Class
public class CustomerEventAssigningJobScheduler
{
private static IScheduler _scheduler;
public static IScheduler scheduler
{
get
{
if (_scheduler == null)
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler().Result;
_scheduler = scheduler;
}
return _scheduler;
}
}
public static async Task Start()
{
await scheduler.Start();
IJobDetail job = JobBuilder.Create<CustomerEventAssigningJob>().WithIdentity("CustomerEventAssigningJob").Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("CustomerEventAssigningJob")
.WithDailyTimeIntervalSchedule
(s =>
s.OnEveryDay()
.StartingDailyAt(TimeOfDay.HourAndMinuteOfDay(00, 10))
.WithIntervalInHours(24)
.InTimeZone(TimeZoneInfo.Utc)
)
.Build();
await scheduler.ScheduleJob(job, trigger);
}
}
I have Tried: .EndingDailyAfterCount(1)
But after adding it does not trigger
screen shot of log is here:
Please check below, as we do not need to use OnEveryDay(), as we already used WithIntervalInHours(24).
ITrigger trigger = TriggerBuilder.Create()
.WithDailyTimeIntervalSchedule
(s =>
s.WithIntervalInHours(24)
.StartingDailyAt(TimeOfDay.HourAndMinuteOfDay(00, 10))
)
.Build();
Its need to check server configuration/deployment, because on local machine, its ran one time.
I was wondering if Quartz.Net has a way to tell (or better write in the logs) on the next execution... I mean, I have got a job that runs at 10:00 AM and it's scheduled to run every two hours... is there a way I can write something as Next run on 12:00AM? or do I have to parse the cron expression, then add it to the current date?
Thanks in advance
Not sure what logger you are using, but this approach should work with any logger that you can create instances of. Create instance of logger and pass it to job via JobDataMap and then use it inside of job. IJobExecutionContext.NextFireTimeUtc will tell you next execution time which you can write to a logger
using System;
using System.Threading.Tasks;
using NLog;
using Quartz;
using Quartz.Impl;
namespace QuartzSampleApp
{
public class Program
{
private static async Task Main(string[] args)
{
var logger = LogManager.GetCurrentClassLogger();
StdSchedulerFactory factory = new StdSchedulerFactory();
IScheduler scheduler = await factory.GetScheduler();
await scheduler.Start();
IJobDetail job = JobBuilder.Create<HelloJob>()
.WithIdentity("job1", "group1")
.Build();
job.JobDataMap["logger"] = logger; // add logger to job data map
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartNow()
.WithSimpleSchedule(x => x
.WithIntervalInSeconds(10)
.RepeatForever())
.Build();
await scheduler.ScheduleJob(job, trigger);
await Task.Delay(TimeSpan.FromSeconds(60));
await scheduler.Shutdown();
Console.WriteLine("Press any key to close the application");
Console.ReadKey();
}
// simple log provider to get something to the console
}
public class HelloJob : IJob
{
public async Task Execute(IJobExecutionContext context)
{
await Console.Out.WriteLineAsync("Greetings from HelloJob!");
var logger = context.JobDetail.JobDataMap["logger"] as ILogger;
logger.Log(LogLevel.Info, "Next job execution at " + context.NextFireTimeUtc);
}
}
}
I have a C# service and I need to run a function once a week.
I have a working C# service which currently is running on a timer every 60 seconds.
Please see below a section of the services OnStart function:
// Set up a timer to trigger.
System.Timers.Timer timer = new System.Timers.Timer
{
Interval = 60000 //*1000; // 60 second
};
timer.Elapsed += delegate {
// Runs the code every 60 seconds but only triggers it if the schedule matches
Function1();
};
timer.Start();
The above code calls Function1() every 60 seconds and I am checking it in Function1 if the current dayofweek and time matches the schedule and if it does than execute the rest of the function.
Although this does work it not the most elegant way IMO.
I have tried using Quartz.net as it was looking promising but when I used all the examples available online (questions answered some 7 years ago in 2012), it is showing as an error in visual studio:
using System;
using Quartz;
public class SimpleJob : IJob
{
public void Execute(IJobExecutionContext context)
{
throw new NotImplementedException();
}
}
This is erroring
(Error CS0738 'SimpleJob' does not implement interface member 'IJob.Execute(IJobExecutionContext)'. 'SimpleJob.Execute(IJobExecutionContext)' cannot implement 'IJob.Execute(IJobExecutionContext)' because it does not have the matching return type of 'Task'.)
but this does not:
public Task Execute(IJobExecutionContext context)
{
throw new NotImplementedException();
}
Could someone give a current working example of a job scheduled through Quartz.net for a beginner?
Or using another elegant method than Quartz.net in a C# service?
First of all we need to implement a job implementation. For example:
internal class TestJob : IJob
{
public Task Execute(IJobExecutionContext context)
{
Console.WriteLine("Job started");
return Task.CompletedTask;
}
}
Now we need to write a method which will return a Scheduler of Quartz :
static async Task TestScheduler()
{
// construct a scheduler factory
NameValueCollection props = new NameValueCollection
{
{ "quartz.serializer.type", "binary" }
};
StdSchedulerFactory factory = new StdSchedulerFactory(props);
// get a scheduler
IScheduler sched = await factory.GetScheduler();
await sched.Start();
// define the job and tie it to our HelloJob class
IJobDetail job = JobBuilder.Create<TestJob>()
.WithIdentity("myJob", "group1")
.Build();
// Trigger the job to run now, and then every 40 seconds
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("myTrigger", "group1")
.StartNow()
.WithSimpleSchedule(x => x
.WithIntervalInMinutes(1)
.RepeatForever())
.Build();
await sched.ScheduleJob(job, trigger);
}
and in the Main method of the Program we will need to write following code:
static async Task Main()
{
Console.WriteLine("Test Scheduler started");
await TestScheduler();
Console.ReadKey();
}
Now this will keep executing after every minute.
Hope it helps.
I have an application in .Net framework and I'm using quartz scheduler. I need to configure quartz.
Now I have one method which is fired every 15 minutes. These method is used to do some work with database. I want, in case, that work of procedure is complete, then start waiting period and after that period again start these database method.
For procedure there will be maximum time which cannot be longer. For examplpe 60 minutes. Do you have any ideas how to configure length of working procedure, how to stop when work is finished and how to define waiting time between?
// configure Quartz
var stdSchedulerProperties = new NameValueCollection
{
{ "quartz.threadPool.threadCount", "10" },
{ "quartz.jobStore.misfireThreshold", "60000" }
};
var stdSchedulerFactory = new StdSchedulerFactory(stdSchedulerProperties);
var scheduler = stdSchedulerFactory.GetScheduler().Result;
scheduler.Start();
// create job and specify timeout
IJobDetail job = JobBuilder.Create<JobWithTimeout>()
.WithIdentity("job1", "group1")
.UsingJobData("timeoutInMinutes", 60)
.Build();
// create trigger and specify repeat interval
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartNow()
.WithSimpleSchedule(x => x.WithIntervalInMinutes(15).RepeatForever())
.Build();
// schedule job
scheduler.ScheduleJob(job, trigger).Wait();
/// <summary>
/// Implementation of IJob. Represents the wrapper job for a task with timeout
/// </summary>
public class JobWithTimeout : IJob
{
public Task Execute(IJobExecutionContext context)
{
return Task.Run(() => Execute(context));
}
public void Execute(IJobExecutionContext context)
{
Thread workerThread = new Thread(DoWork);
workerThread.Start();
context.JobDetail.JobDataMap.TryGetValue("timeoutInMinutes", out object timeoutInMinutes);
TimeSpan timeout = TimeSpan.FromMinutes((int)timeoutInMinutes);
bool finished = workerThread.Join(timeout);
if (!finished) workerThread.Abort();
}
public void DoWork()
{
// do stuff
}
}
I am currently investigating using Quartz.NET for scheduling tasks in my system. As an example of how I am using Quartz.NET, below is a very simple example demonstrating how I am scheduling a task:
class Program
{
static void Main(string[] args)
{
var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "TestScheduler";
properties["quartz.scheduler.instanceId"] = "instance_one";
properties["quartz.jobStore.type"] = "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz";
properties["quartz.jobStore.useProperties"] = "true";
properties["quartz.jobStore.dataSource"] = "default";
properties["quartz.jobStore.tablePrefix"] = "QRTZ_";
properties["quartz.jobStore.lockHandler.type"] = "Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz";
properties["quartz.dataSource.default.connectionString"] = "Server=.\\SqlExpress;Database=quartz;Trusted_Connection=True;";
properties["quartz.dataSource.default.provider"] = "SqlServer-20";
var scheduler = new StdSchedulerFactory(properties).GetScheduler();
scheduler.Start();
TriggerSimpleJob(scheduler);
Console.WriteLine("Waiting For Job");
Console.ReadLine();
}
private static void TriggerSimpleJob(IScheduler scheduler)
{
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartAt(DateBuilder.EvenSecondDateAfterNow())
.UsingJobData("myTriggerParameter", "myTriggerValue")
.UsingJobData("myParameter", "triggerParameter")
.Build();
IJobDetail jobDetail = JobBuilder.Create<SimpleJob>().WithIdentity("job1", "group1")
.UsingJobData("myParameter", "myValue")
.Build();
scheduler.ScheduleJob(jobDetail, trigger);
}
}
public class SimpleJob : IJob
{
public void Execute(IJobExecutionContext context)
{
Console.WriteLine("Job completed");
}
}
The question I have is this:
I would like to decouple of the scheduling of jobs from the execution of jobs.
In the above example, after the job has been scheduled, if the process is still running when the scheduled time arrives the job is executing within this process. Ideally I would like to be able to have a dedicated server with an instance of the Quartz.NET scheduler running that is dedicated to executing jobs, and be able to schedule jobs from other processes knowing the job will be executed on this dedicated server.
I have tried simply setting the property "quartz.threadPool.threadCount" to "0" on the process that schedules jobs, but this throws an exception. Is there any configuration properties on the scheduler that will achieve what I am trying to do?
Good morning,
you can read my answer here.
What I would suggest is to use ADO.NET Job Store (and it seems you're using it).
The application in charge of scheduling jobs should be configured setting the property threadPool to ZeroSizeThreadPool:
properties["quartz.threadPool.type"] = "Quartz.Simpl.ZeroSizeThreadPool, Quartz";
You can read more about this type of thread-pool here.
the application in charge of the execution of the jobs should be configured with these settings:
properties["quartz.threadPool.type"] = "Quartz.Simpl.SimpleThreadPool, Quartz";
properties["quartz.threadPool.threadCount"] = "10";
properties["quartz.threadPool.threadPriority"] = "Normal";
Remove the line
scheduler.Start();