I'm trying to make a generic implementation of kafka consumers in .NET. Basically I don't want to have a consumer for each and every type of message. I want a generic consumer and then the logic of handling the message is in actual Handlers.
What I tried:
Create a generic interface for handlers
public interface IKafkaHandler<TKey, TMessage>
{
Task HandleAsync(TKey key, TMessage message);
}
Create a concrete handler
public class ClubMessageHandler : IKafkaHandler<string, ClubMessage>
{
private readonly ILogger _logger;
public ClubMessageHandler(ILogger logger)
{
_logger = logger;
}
public Task HandleAsync(string key, ClubMessage message)
{
_logger.LogInformation($"Let's go {message.Name}");
return Task.CompletedTask;
}
}
Create a generic consumer
public class GenericConsumer<TKey, TMessage> : BackgroundService
where TKey: class
where TMessage : class
{
private readonly IKafkaHandler<TKey, TMessage> _handler;
public GenericConsumer(IKafkaHandler<TKey, TMessage> handler)
{
_handler = handler;
}
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
var conf = new ConsumerConfig
{
GroupId = "st_consumer_group",
BootstrapServers = "localhost:9092",
AutoOffsetReset = AutoOffsetReset.Earliest
};
using (var builder = new ConsumerBuilder<TKey, TMessage>(conf).Build())
{
builder.Subscribe("topic.name");
var cancelToken = new CancellationTokenSource();
try
{
while (true)
{
var consumer = builder.Consume(stoppingToken);
//here I want to inject the correct handler based on TMessage type
_handler.HandleAsync(consumer.Message.Key, consumer.Message.Value);
}
}
catch (Exception)
{
builder.Close();
}
}
return Task.CompletedTask;
}
}
And this is where I kinda got stuck. If I try in Program.cs
builder.Services.AddSingleton<IKafkaHandler<string, ClubMessage>, ClubMessageHandler>();
I get an exception saying it can't resolve the services (I can post the entire message if needed)
In the end what I want to achieve is to be able to have something like
//Inject Handlers
builder.Services.AddSingleton<IKafkaHandler<string, ClubMessage>, ClubMessageHandler>();
builder.Services.AddSingleton<IKafkaHandler<string, PlayerMessage>, PlayerMessageHandler>();
builder.Services.AddSingleton<IKafkaHandler<string, OtherMessage>, OtherMessageHandler>();
//Start background services
builder.Services.AddSingleton<GenericConsumer<string, ClubMessage>, ClubMessageHandler>();
//etc...
At this point I'm not even sure if this is possible(although it probably is)? I'm not really sure what I'm missing here. What's the correct way to have use dependency injection here?
I have configured Quartz in my .Net program and I have a strange issue happening that I need help with.
What happens is that, if I put the ActuationService with the constructor commented the Quartz implementation works showing the "Hello message. Otherwise, if I have the ActuationService with the constructor method (like in the code below) the message "Hello" does not show
I'm using dependency injection for the Interfaces in the constructor method and I have the following code:
What am I doing wrong ?
using Quartz;
using Quartz.Impl;
using Quartz.Logging;
using Microsoft.Extensions.Logging;
using AAA.Service.Actuations;
namespace AAA.Service.StartUppers
{
public class StartUpService : IStartUpService
{
private readonly IScheduler _quartzScheduler;
public StartUpService(IScheduler quartzScheduler)
{
this._quartzScheduler = quartzScheduler;
}
public async void startTasks()
{
try
{
IJobDetail deviceActuation = JobBuilder.Create<ActuationService>()
.WithIdentity("deviceActuator", "group2")
.Build();
ITrigger triggerDeviceActuation = TriggerBuilder.Create()
.WithIdentity("triggerDeviceActuation", "group2")
.WithCronSchedule("0 0/1 * * * ?")
.Build();
await _quartzScheduler.ScheduleJob(deviceActuation, triggerDeviceActuation);
await _quartzScheduler.Start();
}
catch (Exception e)
{
throw new System.Exception("An error ocurred - " + e.Message);
}
}
}
}
And the class where I have the execute method
namespace AAA.Service.Actuations
{
public class ActuationService : IJob
{
private readonly IUserRepository _userRepo;
private readonly IDeviceRepository _deviceRepo;
private readonly IMeasurementAdapter _measurementsAdapter;
public ActuationService( IUserRepository userRepo, IDeviceRepository deviceRepo, IMeasurementAdapter measurementsAdapter)
{
this._userRepo = userRepo;
this._deviceRepo = deviceRepo;
this._measurementsAdapter = measurementsAdapter;
}
public async Task Execute(IJobExecutionContext context)
{
Console.WriteLine("HELLO");
}
}
}
Qaurtz doesn't know how to instantiate IUserRepository, IDeviceRepository, IMeasurementAdapter. To use DI in Quartz, you must use the Quartz.Extensions.DependencyInjection package.
Call UseMicrosoftDependencyInjectionJobFactory in Quartz configuration:
public void ConfigureServices(IServiceCollection services)
{
//Your services...
services.AddQuartz(q =>
{
q.UseMicrosoftDependencyInjectionJobFactory();
//Your configuration
}
}
Check out the documentation.
Please go through the below scenario. This is based on C# .Net Web API.
I have to call GetMonthlyData() which is located in BL layer. BL Layer related dependencies are initialized in the Controller and it works fine.
It is required to call the same GetMonthlyData() as a Job and I am using Quartz for that. However when initiating the Job I am getting an issue since the dependencies are not resolved at that time since the call does not goes through the controller. How can I resolve this issue?
FileProcessController.cs -> This is the controller which initiates the Dependency Injection
[RoutePrefix("api/FileProcess")]
public class FileProcessController : ApiController
{
private readonly IFileProcessManager _fileProcessManager;
public FileProcessController(IFileProcessManager fileProcessManager)
{
this._fileProcessManager = fileProcessManager;
}
[HttpGet]
[Route("MontlyData")]
public MonthlyProcessDataDM GetMonthlyData()
{
try
{
var result = this._fileProcessManager.GetMonthlyData();
return result;
}
catch (Exception ex)
{
throw ExceptionStatusCode.ThrowHttpErrorResponseAndMessage(ex);
}
}
}
}
FileProcessManager.cs -> This contains the implementation of GetMonthlyData()
public class FileProcessManager : IFileProcessManager
{
public FileProcessManager(IFileProcessManagerRepo ifileProcessManagerRepo)
{
}
public MonthlyProcessDataDM GetMonthlyData()
{
//Code to be executed
}
}
Interface IFileProcessManager
public interface IFileProcessManager
{
MonthlyProcessDataDM GetMonthlyData();
}
Execution of the above flow works properly. In the next scenario, I need to call GetMonthlyData() using Quartz. For that
Global.asax.cs
protected void Application_Start()
{
AreaRegistration.RegisterAllAreas();
System.Web.Http.GlobalConfiguration.Configure(WebApiConfig.Register);
FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
JobScheduler.Start(); // This is the scheduler for the above mentioned job.
}
JobScheduler.cs
public class JobScheduler
{
public static void Start()
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Start();
IJobDetail job = JobBuilder.Create<MonthlyProcessManager>().Build(); // This is the Implementation
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartNow()
.WithSimpleSchedule(x => x
.WithIntervalInSeconds(10)
.RepeatForever())
.Build();
scheduler.ScheduleJob(job, trigger);
}
}
I tried to implement as below to resolve the dependency injection, However it throws null reference exception pointing the dependency injection code.
MonthlyProcessManager.cs
public class MonthlyProcessManager : Quartz.IJob
{
public IFileProcessManager _fileProcessManager;
public MonthlyProcessManager(IFileProcessManager fileProcessManager)
{
_fileProcessManager = fileProcessManager;
}
public void Execute(Quartz.IJobExecutionContext context)
{
MonthlyProcessDataDM monthlyProcessDataVM = _fileProcessManager.GetMonthlyData();
}
}
How can I resolve this issue. Thanks in advance.
I have a console app which uses a class library to execute some long running tasks. This is a .net core console app and uses the .net core Generic Host. I also use the ShellProgressBar library to display some progress bars.
My Hosted service looks like this
internal class MyHostedService : IHostedService, IDisposable
{
private readonly ILogger _logger;
private readonly IMyService _myService;
private readonly IProgress<MyCustomProgress> _progress;
private readonly IApplicationLifetime _appLifetime;
private readonly ProgressBar _progressBar;
private readonly IProgressBarFactory _progressBarFactory;
public MyHostedService(
ILogger<MyHostedService> logger,
IMyService myService,
IProgressBarFactory progressBarFactory,
IApplicationLifetime appLifetime)
{
_logger = logger;
_myService = myService;
_appLifetime = appLifetime;
_progressBarFactory = progressBarFactory;
_progressBar = _progressBarFactory.GetProgressBar(); // this just returns an instance of ShellProgressBar
_progress = new Progress<MyCustomProgress>(progress =>
{
_progressBar.Tick(progress.Current);
});
}
public void Dispose()
{
_progressBar.Dispose();
}
public Task StartAsync(CancellationToken cancellationToken)
{
_myService.RunJobs(_progress);
_appLifetime.StopApplication();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
Where MyCustomProgress looks like this
public class MyCustomProgress
{
public int Current {get; set;}
public int Total {get; set;}
}
and MyService looks something like so (Job1, Job2, Job3 implement IJob)
public class MyService : IMyService
{
private void List<IJob> _jobsToRun;
public MyService()
{
_jobsToRun.Add(new Job1());
_jobsToRun.Add(new Job2());
_jobsToRun.Add(new Job3());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
_jobsToRun.ForEach(job =>
{
job.Execute();
progress.Report(new MyCustomProgress { Current = _jobsToRun.IndexOf(job) + 1, Total = _jobsToRun.Count() });
});
}
}
And IJob is
public interface IJob
{
void Execute();
}
This setup works well and I'm able to display the progress bar from my HostedService by creating a ShellProgressBar instance and using the one IProgress instance I have to update it.
However, I have another implementation of IMyService that I also need to run that looks something like this
public class MyService2 : IMyService
{
private void List<IJob> _sequentialJobsToRun;
private void List<IJob> _parallelJobsToRun;
public MyService()
{
_sequentialJobsToRun.Add(new Job1());
_sequentialJobsToRun.Add(new Job2());
_sequentialJobsToRun.Add(new Job3());
_parallelJobsToRun.Add(new Job4());
_parallelJobsToRun.Add(new Job5());
_parallelJobsToRun.Add(new Job6());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
_sequentialJobsToRun.ForEach(job =>
{
job.Execute();
progress.Report(new MyCustomProgress { Current = _jobsToRun.IndexOf(job) + 1, Total = _jobsToRun.Count() });
});
Parallel.ForEach(_parallelJobsToRun, job =>
{
job.Execute();
// Report progress here
});
}
}
This is the one I'm struggling with. when _parallelJobsToRun is executed, I need to be able to create a new child ShellProgressBar (ShellProgressBar.Spawn) and display them as child progress bars of let's say 'Parallel Jobs'.
This is where I'm looking for some help as to how I can achieve this.
Note: I don't want to take a dependency on ShellProgressBar in my class library containing MyService
Any help much appreciated.
I am a little confused by your description, but let's see if I understand what you are up to. So if you wrap all of this in a class, then taskList1 and taskList2 could be class variables. (By the way taskList1/2 should be named better: say parallelTaskList and whatever . . . anyway.) Then you could write a new method on the class CheckTaskStatus() and just iterate over the two class variables. Does that help or have I completely missed your question?
Can you modify it like this?
public Task<ICollection<IProgress<int>>> StartAsync(CancellationToken cancellationToken)
{
var progressList = _myServiceFromLibrary.RunTasks();
return Task.FromResult(progressList);
}
public ICollection<IProgress<int>> RunTasks()
{
var taskList1 = new List<ITask> { Task1, Task2 };
var plist1 = taskList1.Select(t => t.Progress).ToList();
var taskList2 = new List<ITask> { Task3, Task4, Task5 }:
var plist2 = taskList2.Select(t => t.Progress).ToList();
taskList1.foreach( task => task.Run() );
Parallel.Foreach(taskList2, task => { task.Run() });
return plist1.Concat(plist2).ToList();
}
Task.Progress there is probably a progress getter. realistically IProgress should probably be injected via Tasks constructors. But the point is your public interface doesn't accept list of tasks, thus it should just return collection of progress reports.
How to inject progress reporters into your tasks is a different story that depends on tasks implementations and it may or may not be supported. out of the box.
However what you probably should do is to supply progress callback or progress factory so that progress reporters of your choice are created:
public Task StartAsync(CancellationToken cancellationToken, Action<Task,int> onprogress)
{
_myServiceFromLibrary.RunTasks(onprogress);
return Task.CompletedTask;
}
public class SimpleProgress : IProgress<int>
{
private readonly Task task;
private readonly Action<Task,int> action;
public SimpleProgress(Task task, Action<Task,int> action)
{
this.task = task;
this.action = action;
}
public void Report(int progress)
{
action(task, progress);
}
}
public ICollection<IProgress<int>> RunTasks(Action<Task,int> onprogress)
{
var taskList1 = new List<ITask> { Task1, Task2 };
taskList1.foreach(t => t.Progress = new SimpleProgress(t, onprogress));
var taskList2 = new List<ITask> { Task3, Task4, Task5 }:
taskList2.foreach(t => t.Progress = new SimpleProgress(t, onprogress));
taskList1.foreach( task => task.Run() );
Parallel.Foreach(taskList2, task => { task.Run() });
}
you may see here, that it really is mostly question about how your tasks are going to call IProgress<T>.Report(T value) method.
Honestly I would just use an event in your task prototype.
It's not really clear exactly what you want because the code you posted doesn't match the names you then reference in your question text... It would be helpful to have all the code (the RunTasks function for example, your IProgress prototype, etc).
Nevertheless, an event exists specifically to signal calling code. Let's go back to the basics. Let's say you have library called MyLib, with a method DoThings().
Create a new class that inherits from EventArgs, and that will carry your task's progress reports...
public class ProgressEventArgs : EventArgs
{
private int _taskId;
private int _percent;
private string _message;
public int TaskId => _taskId;
public int Percent => _percent;
public string Message => _message;
public ProgressEventArgs(int taskId, int percent, string message)
{
_taskId = taskId;
_percent = percent;
_message = message;
}
}
Then on your library's class definition, add an event like so:
public event EventHandler<ProgressEventArgs> Progress;
And in your console application, create a handler for progress events:
void ProgressHandler(object sender, ProgressEventArgs e)
{
// Do whatever you want with your progress report here, all your
// info is in the e variable
}
And subscribe to your class library's event:
var lib = new MyLib();
lib.Progress += ProgressHandler;
lib.DoThings();
When you are done, unsubscribe from the event:
lib.Progress -= ProgressHandler;
In your class library, now you can send back progress reports by raising the event in your code. First create a stub method to invoke the event:
protected virtual void OnProgress(ProgressEventArgs e)
{
var handler = Progress;
if (handler != null)
{
handler(this, e);
}
}
And then add this to your task's code where you want it:
OnProgress(new ProgressEventArgs(2452343, 10, "Reindexing google..."));
The only thing to be careful about is to report progress sparingly, because each time your event fires it interrupts your console application, and you can really bog it down hard if you send 10 million events all at once. Be logical about it.
Alternate way; If you own the code IProgress<T> and Progress
IProgress<T>
{
IProgress<T> CreateNew();
Report(T progress);
}
Progress<T> : IProgress<T>
{
Progress(ShellProgressClass)
{
// initialize progressBar or span new
}
....
IProgress<T> CreateNew()
{
return new Progress();
}
}
you can later improvise to have one big progressBar (collection of Sequential or Parallel) and what not
Your MyService could have a dependency similar to:
public interface IJobContainer
{
void Add(IJob job);
void RunJobs(IProgress<MyProgress> progress, Action<IJob>? callback = null); // Using an action for extra work you may want to do
}
This way you don't have to worry about reporting progress in MyService (which doesn't feel like it should be MyService's job anyway. The implementation could look something like this for the parallel job container:
public class MyParallelJobContainer
{
private readonly IList<IJob> parallelJobs = new List<IJob>();
public MyParallelJobContainer()
{
this.progress = progress;
}
public void Add(IJob job) { ... }
void RunJobs(IProgress<MyProgress> progress, Action<IJob>? callback = null)
{
using (var progressBar = new ProgressBar(options...))
{
Parallel.ForEach(parallelJobs, job =>
{
callback?.Invoke(job);
job.Execute();
progressBar.Tick();
})
}
}
}
MyService would then look like this:
public class MyService : IMyService
{
private readonly IJobContainer sequentialJobs;
private readonly IJobContainer parallelJobs;
public MyService(
IJobContainer sequentialJobs,
IJobContainer parallelJobs)
{
this.sequentialJobs = sequentialJobs;
this.parallelJobs = parallelJobs;
this.sequentialJobs.Add(new DoSequentialJob1());
this.sequentialJobs.Add(new DoSequentialJob2());
this.sequentialJobs.Add(new DoSequentialJob3));
this.parallelJobs.Add(new DoParallelJobA());
this.parallelJobs.Add(new DoParallelJobB());
this.parallelJobs.Add(new DoParallelJobC());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
sequentialJobs.RunJobs(progress, job =>
{
// do something with the job if necessary
});
parallelJobs.RunJobs(progress, job =>
{
// do something with the job if necessary
});
}
The advantage of this way is that MyService only has one job and doesn't have to worry about what you do once the job is completed.
From my understanding of your issue the question is how do you display progress across both completion of the synchronous jobs and parallelized jobs.
In theory the parallel jobs could start and finish at the same time, so you could treat the parallel jobs as a single job. Instead of using the count of sequential jobs as your total, increase that number by one. This might be satisfactory for a small number of parallel jobs.
If you want to add progress between the parallel jobs, you will need to handle multi-threading in your code because the parallel jobs will be running concurrently.
object pJobLock = new object();
int numProcessed = 0;
foreach(var parallelJob in parallelJobs)
{
parallelJob.DoWork();
lock (pJobLock)
{
numProcessed++;
progress.Report(new MyCustomProgress { Current = numProcessed, Total = parallelJobs.Count() });
}
}
I have created a generic job schedule manager class. I am using Quartz.net 3.0.7 in .net core app.
public class ScheduleManagerManager<TJob> : IScheduleManager where TJob : IJob
{
private readonly string _cronExpression;
private readonly string _jobIdentity;
private readonly string _triggerIdentity;
private IScheduler _scheduler;
private IJobDetail _jobDetail;
private ITrigger _trigger;
public ScheduleManagerManager(string cronExpression)
{
_cronExpression = cronExpression;
_jobIdentity = Guid.NewGuid().ToString();
_triggerIdentity = Guid.NewGuid().ToString();
}
public async Task Run()
{
try
{
_scheduler = await StdSchedulerFactory.GetDefaultScheduler();
await _scheduler.Start();
_jobDetail = JobBuilder.Create<TJob>()
.WithIdentity(_jobIdentity)
.Build();
_trigger = TriggerBuilder.Create()
.WithIdentity(_triggerIdentity)
.StartNow()
.WithCronSchedule(_cronExpression)
.Build();
await _scheduler.ScheduleJob(_jobDetail, _trigger);
}
catch (SchedulerException se)
{
await Console.Out.WriteLineAsync(se.Message);
}
}
public void ShutDown()
{
_scheduler.Shutdown();
}
}
So I amusing this in applications like following:
var manager1 = new ScheduleManagerManager<MailSender>();
var manager2 = new ScheduleManagerManager<SmsSender>();
I want to add an event handler to get job data after executed the job.
var manager1 = new ScheduleManagerManager<MailSender>();
manager1.JobExecuted += OnJobExecuted();
But there is no event in Quartz.net jobs or triggers. How can I do it?
But there is no event in Quartz.net jobs or triggers. Of course they are, they are called Listener.
Just implement IJobListener:
public class MyJobListener : IJobListener
{
public string Name () => "MyListener";
public Task JobToBeExecuted(IJobExecutionContext context){ }
public Task JobExecutionVetoed(IJobExecutionContext context){ }
public Task JobWasExecuted(IJobExecutionContext context, JobExecutionException jobException)
{
// Do after Job stuff....
}
}
And add it to your scheduler in your ScheduleManagerManager:
var myJobListener = new MyJobListener();
_scheduler.ListenerManager.AddJobListener(myJobListener, GroupMatcher<JobKey>.AnyGroup());