I'm trying to make a generic implementation of kafka consumers in .NET. Basically I don't want to have a consumer for each and every type of message. I want a generic consumer and then the logic of handling the message is in actual Handlers.
What I tried:
Create a generic interface for handlers
public interface IKafkaHandler<TKey, TMessage>
{
Task HandleAsync(TKey key, TMessage message);
}
Create a concrete handler
public class ClubMessageHandler : IKafkaHandler<string, ClubMessage>
{
private readonly ILogger _logger;
public ClubMessageHandler(ILogger logger)
{
_logger = logger;
}
public Task HandleAsync(string key, ClubMessage message)
{
_logger.LogInformation($"Let's go {message.Name}");
return Task.CompletedTask;
}
}
Create a generic consumer
public class GenericConsumer<TKey, TMessage> : BackgroundService
where TKey: class
where TMessage : class
{
private readonly IKafkaHandler<TKey, TMessage> _handler;
public GenericConsumer(IKafkaHandler<TKey, TMessage> handler)
{
_handler = handler;
}
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
var conf = new ConsumerConfig
{
GroupId = "st_consumer_group",
BootstrapServers = "localhost:9092",
AutoOffsetReset = AutoOffsetReset.Earliest
};
using (var builder = new ConsumerBuilder<TKey, TMessage>(conf).Build())
{
builder.Subscribe("topic.name");
var cancelToken = new CancellationTokenSource();
try
{
while (true)
{
var consumer = builder.Consume(stoppingToken);
//here I want to inject the correct handler based on TMessage type
_handler.HandleAsync(consumer.Message.Key, consumer.Message.Value);
}
}
catch (Exception)
{
builder.Close();
}
}
return Task.CompletedTask;
}
}
And this is where I kinda got stuck. If I try in Program.cs
builder.Services.AddSingleton<IKafkaHandler<string, ClubMessage>, ClubMessageHandler>();
I get an exception saying it can't resolve the services (I can post the entire message if needed)
In the end what I want to achieve is to be able to have something like
//Inject Handlers
builder.Services.AddSingleton<IKafkaHandler<string, ClubMessage>, ClubMessageHandler>();
builder.Services.AddSingleton<IKafkaHandler<string, PlayerMessage>, PlayerMessageHandler>();
builder.Services.AddSingleton<IKafkaHandler<string, OtherMessage>, OtherMessageHandler>();
//Start background services
builder.Services.AddSingleton<GenericConsumer<string, ClubMessage>, ClubMessageHandler>();
//etc...
At this point I'm not even sure if this is possible(although it probably is)? I'm not really sure what I'm missing here. What's the correct way to have use dependency injection here?
I created an API in .net core with C# that has a collection of elements. That collection has a delegate that is triggered when a new element is added (is working). The problem is that I can't make it work unless I subscribe to the delegate from the controller.
Current scenario:
[HttpPost("welcome")]
public IActionResult Welcome([FromBody] MyClient newClient)
{
ClientsDataStore.Current.clients.Push(newClient);
return Ok(newClient);
}
I created the class MyClass:
public class MyClass
{
ILogger<MyClass> logger;
public MyClass(ILogger<MyClass> logger)
{
this.logger = logger;
ClientsDataStore.Current.clients.ElementAdded += ListenClients;
}
public async void ListenClients(object sender, MyClient e)
{
logger.LogDebug($"New client added");
if (e.VirtualMachine)
{
await ProcessVMAsync(e);
}
else
{
await ProcessDesktopAsync(e);
}
}
}
I tried to register the class in public void ConfigureServices(IServiceCollection services) like this:
services.AddTransient<MyClass>();
With that scenario, the delegate is never triggered, but if I use:
[HttpPost("welcome")]
public IActionResult Welcome([FromBody] MyClient newClient)
{
ClientsDataStore.Current.clients.ElementAdded += ListenClients;
ClientsDataStore.Current.clients.Push(newClient);
return Ok(newClient);
}
Then the event is triggered.
I would like to know what I am missing here and how could I accomplished this.
UPDATE:
I changed the approach of the solution. I am going to have a background service checking every 10 seconds if the collection has a new element.
I have a console app which uses a class library to execute some long running tasks. This is a .net core console app and uses the .net core Generic Host. I also use the ShellProgressBar library to display some progress bars.
My Hosted service looks like this
internal class MyHostedService : IHostedService, IDisposable
{
private readonly ILogger _logger;
private readonly IMyService _myService;
private readonly IProgress<MyCustomProgress> _progress;
private readonly IApplicationLifetime _appLifetime;
private readonly ProgressBar _progressBar;
private readonly IProgressBarFactory _progressBarFactory;
public MyHostedService(
ILogger<MyHostedService> logger,
IMyService myService,
IProgressBarFactory progressBarFactory,
IApplicationLifetime appLifetime)
{
_logger = logger;
_myService = myService;
_appLifetime = appLifetime;
_progressBarFactory = progressBarFactory;
_progressBar = _progressBarFactory.GetProgressBar(); // this just returns an instance of ShellProgressBar
_progress = new Progress<MyCustomProgress>(progress =>
{
_progressBar.Tick(progress.Current);
});
}
public void Dispose()
{
_progressBar.Dispose();
}
public Task StartAsync(CancellationToken cancellationToken)
{
_myService.RunJobs(_progress);
_appLifetime.StopApplication();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
Where MyCustomProgress looks like this
public class MyCustomProgress
{
public int Current {get; set;}
public int Total {get; set;}
}
and MyService looks something like so (Job1, Job2, Job3 implement IJob)
public class MyService : IMyService
{
private void List<IJob> _jobsToRun;
public MyService()
{
_jobsToRun.Add(new Job1());
_jobsToRun.Add(new Job2());
_jobsToRun.Add(new Job3());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
_jobsToRun.ForEach(job =>
{
job.Execute();
progress.Report(new MyCustomProgress { Current = _jobsToRun.IndexOf(job) + 1, Total = _jobsToRun.Count() });
});
}
}
And IJob is
public interface IJob
{
void Execute();
}
This setup works well and I'm able to display the progress bar from my HostedService by creating a ShellProgressBar instance and using the one IProgress instance I have to update it.
However, I have another implementation of IMyService that I also need to run that looks something like this
public class MyService2 : IMyService
{
private void List<IJob> _sequentialJobsToRun;
private void List<IJob> _parallelJobsToRun;
public MyService()
{
_sequentialJobsToRun.Add(new Job1());
_sequentialJobsToRun.Add(new Job2());
_sequentialJobsToRun.Add(new Job3());
_parallelJobsToRun.Add(new Job4());
_parallelJobsToRun.Add(new Job5());
_parallelJobsToRun.Add(new Job6());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
_sequentialJobsToRun.ForEach(job =>
{
job.Execute();
progress.Report(new MyCustomProgress { Current = _jobsToRun.IndexOf(job) + 1, Total = _jobsToRun.Count() });
});
Parallel.ForEach(_parallelJobsToRun, job =>
{
job.Execute();
// Report progress here
});
}
}
This is the one I'm struggling with. when _parallelJobsToRun is executed, I need to be able to create a new child ShellProgressBar (ShellProgressBar.Spawn) and display them as child progress bars of let's say 'Parallel Jobs'.
This is where I'm looking for some help as to how I can achieve this.
Note: I don't want to take a dependency on ShellProgressBar in my class library containing MyService
Any help much appreciated.
I am a little confused by your description, but let's see if I understand what you are up to. So if you wrap all of this in a class, then taskList1 and taskList2 could be class variables. (By the way taskList1/2 should be named better: say parallelTaskList and whatever . . . anyway.) Then you could write a new method on the class CheckTaskStatus() and just iterate over the two class variables. Does that help or have I completely missed your question?
Can you modify it like this?
public Task<ICollection<IProgress<int>>> StartAsync(CancellationToken cancellationToken)
{
var progressList = _myServiceFromLibrary.RunTasks();
return Task.FromResult(progressList);
}
public ICollection<IProgress<int>> RunTasks()
{
var taskList1 = new List<ITask> { Task1, Task2 };
var plist1 = taskList1.Select(t => t.Progress).ToList();
var taskList2 = new List<ITask> { Task3, Task4, Task5 }:
var plist2 = taskList2.Select(t => t.Progress).ToList();
taskList1.foreach( task => task.Run() );
Parallel.Foreach(taskList2, task => { task.Run() });
return plist1.Concat(plist2).ToList();
}
Task.Progress there is probably a progress getter. realistically IProgress should probably be injected via Tasks constructors. But the point is your public interface doesn't accept list of tasks, thus it should just return collection of progress reports.
How to inject progress reporters into your tasks is a different story that depends on tasks implementations and it may or may not be supported. out of the box.
However what you probably should do is to supply progress callback or progress factory so that progress reporters of your choice are created:
public Task StartAsync(CancellationToken cancellationToken, Action<Task,int> onprogress)
{
_myServiceFromLibrary.RunTasks(onprogress);
return Task.CompletedTask;
}
public class SimpleProgress : IProgress<int>
{
private readonly Task task;
private readonly Action<Task,int> action;
public SimpleProgress(Task task, Action<Task,int> action)
{
this.task = task;
this.action = action;
}
public void Report(int progress)
{
action(task, progress);
}
}
public ICollection<IProgress<int>> RunTasks(Action<Task,int> onprogress)
{
var taskList1 = new List<ITask> { Task1, Task2 };
taskList1.foreach(t => t.Progress = new SimpleProgress(t, onprogress));
var taskList2 = new List<ITask> { Task3, Task4, Task5 }:
taskList2.foreach(t => t.Progress = new SimpleProgress(t, onprogress));
taskList1.foreach( task => task.Run() );
Parallel.Foreach(taskList2, task => { task.Run() });
}
you may see here, that it really is mostly question about how your tasks are going to call IProgress<T>.Report(T value) method.
Honestly I would just use an event in your task prototype.
It's not really clear exactly what you want because the code you posted doesn't match the names you then reference in your question text... It would be helpful to have all the code (the RunTasks function for example, your IProgress prototype, etc).
Nevertheless, an event exists specifically to signal calling code. Let's go back to the basics. Let's say you have library called MyLib, with a method DoThings().
Create a new class that inherits from EventArgs, and that will carry your task's progress reports...
public class ProgressEventArgs : EventArgs
{
private int _taskId;
private int _percent;
private string _message;
public int TaskId => _taskId;
public int Percent => _percent;
public string Message => _message;
public ProgressEventArgs(int taskId, int percent, string message)
{
_taskId = taskId;
_percent = percent;
_message = message;
}
}
Then on your library's class definition, add an event like so:
public event EventHandler<ProgressEventArgs> Progress;
And in your console application, create a handler for progress events:
void ProgressHandler(object sender, ProgressEventArgs e)
{
// Do whatever you want with your progress report here, all your
// info is in the e variable
}
And subscribe to your class library's event:
var lib = new MyLib();
lib.Progress += ProgressHandler;
lib.DoThings();
When you are done, unsubscribe from the event:
lib.Progress -= ProgressHandler;
In your class library, now you can send back progress reports by raising the event in your code. First create a stub method to invoke the event:
protected virtual void OnProgress(ProgressEventArgs e)
{
var handler = Progress;
if (handler != null)
{
handler(this, e);
}
}
And then add this to your task's code where you want it:
OnProgress(new ProgressEventArgs(2452343, 10, "Reindexing google..."));
The only thing to be careful about is to report progress sparingly, because each time your event fires it interrupts your console application, and you can really bog it down hard if you send 10 million events all at once. Be logical about it.
Alternate way; If you own the code IProgress<T> and Progress
IProgress<T>
{
IProgress<T> CreateNew();
Report(T progress);
}
Progress<T> : IProgress<T>
{
Progress(ShellProgressClass)
{
// initialize progressBar or span new
}
....
IProgress<T> CreateNew()
{
return new Progress();
}
}
you can later improvise to have one big progressBar (collection of Sequential or Parallel) and what not
Your MyService could have a dependency similar to:
public interface IJobContainer
{
void Add(IJob job);
void RunJobs(IProgress<MyProgress> progress, Action<IJob>? callback = null); // Using an action for extra work you may want to do
}
This way you don't have to worry about reporting progress in MyService (which doesn't feel like it should be MyService's job anyway. The implementation could look something like this for the parallel job container:
public class MyParallelJobContainer
{
private readonly IList<IJob> parallelJobs = new List<IJob>();
public MyParallelJobContainer()
{
this.progress = progress;
}
public void Add(IJob job) { ... }
void RunJobs(IProgress<MyProgress> progress, Action<IJob>? callback = null)
{
using (var progressBar = new ProgressBar(options...))
{
Parallel.ForEach(parallelJobs, job =>
{
callback?.Invoke(job);
job.Execute();
progressBar.Tick();
})
}
}
}
MyService would then look like this:
public class MyService : IMyService
{
private readonly IJobContainer sequentialJobs;
private readonly IJobContainer parallelJobs;
public MyService(
IJobContainer sequentialJobs,
IJobContainer parallelJobs)
{
this.sequentialJobs = sequentialJobs;
this.parallelJobs = parallelJobs;
this.sequentialJobs.Add(new DoSequentialJob1());
this.sequentialJobs.Add(new DoSequentialJob2());
this.sequentialJobs.Add(new DoSequentialJob3));
this.parallelJobs.Add(new DoParallelJobA());
this.parallelJobs.Add(new DoParallelJobB());
this.parallelJobs.Add(new DoParallelJobC());
}
public void RunJobs(IProgress<MyCustomProgress> progress)
{
sequentialJobs.RunJobs(progress, job =>
{
// do something with the job if necessary
});
parallelJobs.RunJobs(progress, job =>
{
// do something with the job if necessary
});
}
The advantage of this way is that MyService only has one job and doesn't have to worry about what you do once the job is completed.
From my understanding of your issue the question is how do you display progress across both completion of the synchronous jobs and parallelized jobs.
In theory the parallel jobs could start and finish at the same time, so you could treat the parallel jobs as a single job. Instead of using the count of sequential jobs as your total, increase that number by one. This might be satisfactory for a small number of parallel jobs.
If you want to add progress between the parallel jobs, you will need to handle multi-threading in your code because the parallel jobs will be running concurrently.
object pJobLock = new object();
int numProcessed = 0;
foreach(var parallelJob in parallelJobs)
{
parallelJob.DoWork();
lock (pJobLock)
{
numProcessed++;
progress.Report(new MyCustomProgress { Current = numProcessed, Total = parallelJobs.Count() });
}
}
So far I could make a delegate type, for example:
// Can't use Task in WinRT interface and TypedEventHandler doesn't work with async await
public delegate IAsyncOperation<string> AsyncEventHandler(object sender, object args);
And then expose in WinRT object like so:
public AsyncEventHandler OnMyEvent { get; set; }
In the WinRT object I would call it like this:
if (OnMyEvent != null)
{
var result = await OnMyEvent.Invoke(this, someArgs);
// Do something with the result...
}
And the client code consuming the WinRT object could do this:
instanceOfWinRTObject.OnMyEvent = OnCalledBackFromWinRT;
But because the delegate returns an IAsyncOperation we need to do some wrapping:
private async Task<string> OnCalledBackFromWinRTAsync(object sender,
object args)
{
return await GetSomeStringAsync(args);
}
private IAsyncOperation<string> OnCalledBackFromWinRT(object sender, object args)
{
return OnCalledBackFromWinRTAsync(sender, args).AsAsyncOperation();
}
It just feels like there must be a cleaner way to achieve this.
Here is an alternative prompted by Peter Torr's comment.
// Custom event args class
public sealed class MyEventArgs
{
public string Result;
public Deferral Deferral;
}
// Declare the event handler on the WinRT component
public event TypedEventHandler<object, MyEventArgs> OnSuspendingEvent;
Then in the WinRT component you could invoke the event like this:
if (OnSuspendingEvent != null)
{
var tcs = new TaskCompletionSource();
using (var deferral = new Deferral(() => tcs.SetResult(true)))
{
var myArgs = MyEventArgs();
myArgs.Deferral = deferral;
OnSuspendUnloading.Invoke(this, myArgs);
await tcs.Task;
DoSomethingWithResult(args.Result);
}
}
And finally in the client code add a handler like so:
instanceOfWinRTObject.OnSuspendingEvent += async (sender, args) =>
{
var deferral = args.Deferral;
try
{
// Client does some async work with a result
args.Result = await GetSomeStringAsync(args);
}
finally
{
deferral.Complete();
}
}
I have a service that exposes async operation via event driven async pattern.
public interface IService
{
void DoAsync(int param);
event DoCompleted;
}
There is another class that depends on IService service object
public class Foo
{
private IService _service;
public EventHandler CalculationComplete;
public void Foo(IService service) {_service = service};
public int Calculated;
public void CalculateAsync(int param)
{
//Invoke _service.DoAsync(param)
//(...)
}
}
Basically after calling foo.CalculateAsyc CalculationComplete should notify consumer of calc completion.
The question is how to mock IService when unit testing Foo ? I am using Moq. More specifically how to make unittest wait for CalculationComplete event and react accordingly?
Hard to know what you are trying to test here, so I can't give you a 100% accurate sample. Your sample code seems to be missing quite a few details... I filled some of the missing bits in, but there are more questions.
In any case, a method I use for waiting on events is a semaphore. I like to use AutoResetEvent for this in simple occasions like this.
public class Foo
{
private IService _service;
public EventHandler CalculationComplete;
public Foo(IService service)
{
_service = service;
_service.DoCompleted += (o,e) =>
{
Calculated = e.Result;
if(CalculationComplete != null) { CalculationComplete(this, new EventArgs()); }
};
}
public int Calculated;
public void CalculateAsync(int param)
{
_service.DoAsync(param);
}
}
public interface IService
{
void DoAsync(int param);
event EventHandler<DoResultEventArgs> DoCompleted;
}
public class DoResultEventArgs : EventArgs
{
public int Result { get; set; }
}
[TestMethod]
public void CalculateAsync_CallsService_CalculatedIsPopulated()
{
//Arrange
Mock<IService> sMock = new Mock<IService>();
sMock.Setup(s => s.DoAsync(It.IsAny<int>()))
.Raises(s => s.DoCompleted += null, new DoResultEventArgs() { Result = 324 });
Foo foo = new Foo(sMock.Object);
AutoResetEvent waitHandle = new AutoResetEvent(false);
foo.CalculationComplete += (o,e) => waitHandle.Set();
//Act
foo.CalculateAsync(12);
waitHandle.WaitOne();
//Assert
Assert.IsEqual(foo.Calculated, 324);
}
Without more information, this is the best I can do. I hope it was what you were looking for.