Kafka consume message and then produce to another topic - c#

I have to consume from a Kafka topic, get the message and do some json clean and filter job, then I need to produce the new message to another Kafka topic, my code is like this:
public static YamlMappingNode configs;
public static void Main(string[] args)
{
using (var reader = new StreamReader(Path.Combine(Directory.GetCurrentDirectory(), ".gitlab-ci.yml")))
{
var yaml = new YamlStream();
yaml.Load(reader);
//find variables
configs = (YamlMappingNode)yaml.Documents[0].RootNode;
configs = (YamlMappingNode)configs.Children.Where(k => k.Key.ToString() == "variables")?.FirstOrDefault().Value;
}
CancellationTokenSource cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => {
e.Cancel = true; // prevent the process from terminating.
cts.Cancel();
};
Run_ManualAssign(configs, cts.Token);
}
public static async void Run_ManualAssign(YamlMappingNode configs, CancellationToken cancellationToken)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topics = configs.Where(k => k.Key.ToString() == "input_kfk_topic")?.FirstOrDefault().Value.ToString();
var config = new ConsumerConfig
{
// the group.id property must be specified when creating a consumer, even
// if you do not intend to use any consumer group functionality.
GroupId = new Guid().ToString(),
BootstrapServers = brokerList,
// partition offsets can be committed to a group even by consumers not
// subscribed to the group. in this example, auto commit is disabled
// to prevent this from occurring.
EnableAutoCommit = true
};
using (var consumer =
new ConsumerBuilder<Ignore, string>(config)
.SetErrorHandler((_, e) => Console.WriteLine($"Error: {e.Reason}"))
.Build())
{
//consumer.Assign(topics.Select(topic => new TopicPartitionOffset(topic, 0, Offset.Beginning)).ToList());
consumer.Assign(new TopicPartitionOffset(topics, 0, Offset.End));
//var producer = new ProducerBuilder<Null, string>(config).Build();
try
{
while (true)
{
try
{
var consumeResult = consumer.Consume(cancellationToken);
/// Note: End of partition notification has not been enabled, so
/// it is guaranteed that the ConsumeResult instance corresponds
/// to a Message, and not a PartitionEOF event.
//filter message
var result = ReadMessage(configs, consumeResult.Message.Value);
//send to kafka topic
await Run_ProducerAsync(configs, result);
}
catch (ConsumeException e)
{
Console.WriteLine($"Consume error: {e.Error.Reason}");
}
}
}
catch (OperationCanceledException)
{
Console.WriteLine("Closing consumer.");
consumer.Close();
}
}
}
#endregion
#region Run_Producer
public static async Task Run_ProducerAsync(YamlMappingNode configs, string message)
{
var brokerList = configs.Where(k => k.Key.ToString() == "kfk_broker")?.FirstOrDefault().Value.ToString();
var topicName = configs.Where(k => k.Key.ToString() == "target_kafka_topic")?.FirstOrDefault().Value.ToString();
var config = new ProducerConfig {
BootstrapServers = brokerList,
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
try
{
/// Note: Awaiting the asynchronous produce request below prevents flow of execution
/// from proceeding until the acknowledgement from the broker is received (at the
/// expense of low throughput).
var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });
producer.Flush(TimeSpan.FromSeconds(10));
Console.WriteLine($"delivered to: {deliveryReport.TopicPartitionOffset}");
}
catch (ProduceException<string, string> e)
{
Console.WriteLine($"failed to deliver message: {e.Message} [{e.Error.Code}]");
}
}
}
#endregion
Am I doing something wrong here? The program existed immediately when executing var deliveryReport = await producer.ProduceAsync(topicName, new Message<Null, string> { Value = message });, no error message, no error code.
In the meanwhile I used Python and config the same for Producer, it works well.

Run_ManualAssign(configs, cts.Token);
For this line in the Main function, you are calling async without await in a sync function. Thus the program exit immediately after this invoke started (not finished as it is async)
You could have 2 options
Use async Main function and add await in front of this invoke.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-7.1/async-main
If you really want to call async function in sync function
Run_ManualAssign(configs, ts.Token).ConfigureAwait(false).GetAwaiter().GetResult();

I solved this problem but I don't know why actually. I opened an issue here.

Related

Correct way of starting a forever-running loop with Async

I've an existing code I wrote some time ago, that works but I dislike the fact that the thread I start remains in loop.
This piece of code is a consumer on an IBMMQ code, waiting for messages to be processed.The problem I've is that with the following code
private Task ExecuteQueuePolling(CancellationToken cancellationToken)
{
return Task.Factory.StartNew(() =>
{
ConnectToAccessQueue();
Logger.Debug($"Accessed to the queue {queueName}");
Logger.DebugFormat("Repeating timer started, checking frequency: {checkingFrequency}",
checkingFrequency);
while (!cancellationToken.IsCancellationRequested)
{
Logger.Trace( () => "Listening on queues for new messages");
// isChecking = true;
var mqMsg = new MQMessage();
var mqGetMsgOpts = new MQGetMessageOptions
{ WaitInterval = (int)checkingFrequency.TotalMilliseconds };
// 15 second limit for waiting
mqGetMsgOpts.Options |= MQC.MQGMO_WAIT | MQC.MQGMO_FAIL_IF_QUIESCING |
MQC.MQCNO_RECONNECT_Q_MGR | MQC.MQOO_INPUT_AS_Q_DEF;
try
{
mqQueue.Get(mqMsg, mqGetMsgOpts);
if (string.Compare(mqMsg.Format, MQC.MQFMT_STRING, StringComparison.Ordinal) == 0)
{
var text = mqMsg.ReadString(mqMsg.MessageLength);
Logger.Debug($"Message received : [{text}]");
Message message = new Message { Content = text };
foreach (var observer in observers)
observer.OnNext(message);
}
else
{
Logger.Warn("Non-text message");
}
}
catch (MQException ex)
{
if (ex.Message == MQC.MQRC_NO_MSG_AVAILABLE.ToString())
{
Logger.Trace("No messages available");
//nothing to do, emtpy queue
}
else if (ex.Message == MQC.MQRC_CONNECTION_BROKEN.ToString())
{
Logger.ErrorException("MQ Exception, trying to recconect", ex);
throw new ReconnectException();
}
}
Thread.Sleep(100);
}
},cancellationToken);
}
//Calling method
try
{
string queueManagerName = configuration.GetValue<string>("IBMMQ:QUEUE_MANAGER_NAME");
// var queueManager = new MQQueueManager(queueManagerName,dictionary2);
QueueMonitor monitor = new QueueMonitor(configuration, "IMPORTER_RECEIVER_TEST");
//_subscription = monitor.Subscribe(receiver);
await monitor.StartAsync(cts.Token).ConfigureAwait(false);
}
catch (Exception e)
{
log.Error(e, "Error creating the queue monitor or it's subscription");
}
finally
{
WaitForCancel(cts);
}
The call to await monitor.StartAsync(cts.Token).ConfigureAwait(false); remains pending.
How should I modify my code, so that the call returns and in background the task continue to loop?
Thanks in advance
Here is how you can simplify your code by replacing Thread.Sleep with Task.Delay:
private async Task ExecuteQueuePolling(CancellationToken cancellationToken)
{
while (true)
{
// Process mqQueue here
await Task.Delay(100, cancellationToken);
}
}
Task.Delay has the advantage that accepts a CancellationToken, so in case of cancellation the loop will exit immediately. This could be important if the pooling of the MQ was lazier (for example every 5 seconds).
private static Task _runningTask;
static void Main(string[] args)
{
var cts = new CancellationTokenSource();
_runningTask = ExecuteQueuePolling(cts.Token);
WaitForCancel(cts);
}
private static void WaitForCancel(CancellationTokenSource cts)
{
var spinner = new SpinWait();
while (!cts.IsCancellationRequested
&& _runningTask.Status == TaskStatus.Running) spinner.SpinOnce();
}
private static Task ExecuteQueuePolling(CancellationToken cancellationToken)
{
var t = new Task(() =>
{
while (!cancellationToken.IsCancellationRequested)
; // your code
if (cancellationToken.IsCancellationRequested)
throw new OperationCanceledException();
}, cancellationToken, TaskCreationOptions.LongRunning);
t.Start();
return t;
}

Problems working with async Task and Textbox.Text = "Hello"

First of all, sorry because I am so new at C# and I decided to make this question because I have been choked in this for hours.
I have an GUI that works with Google Cloud Speech services and make a Speech-to-Text operation. I share with you the whole method that runs when a button is clicked:
private async Task<object> StreamingMicRecognizeAsync(int seconds)
{
if (NAudio.Wave.WaveIn.DeviceCount < 1)
{
Console.WriteLine("No microphone!");
return -1;
}
GoogleCredential googleCredential;
using (Stream m = new FileStream(#"..\..\credentials.json", FileMode.Open))
googleCredential = GoogleCredential.FromStream(m);
var channel = new Grpc.Core.Channel(SpeechClient.DefaultEndpoint.Host,
googleCredential.ToChannelCredentials());
var speech = SpeechClient.Create(channel);
var streamingCall = speech.StreamingRecognize();
// Write the initial request with the config.
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
StreamingConfig = new StreamingRecognitionConfig()
{
Config = new RecognitionConfig()
{
Encoding =
RecognitionConfig.Types.AudioEncoding.Linear16,
SampleRateHertz = 48000,
LanguageCode = "es-ES",
},
InterimResults = true,
}
});
// Read from the microphone and stream to API.
object writeLock = new object();
bool writeMore = true;
var waveIn = new NAudio.Wave.WaveInEvent();
waveIn.DeviceNumber = 0;
waveIn.WaveFormat = new NAudio.Wave.WaveFormat(48000, 1);
waveIn.DataAvailable +=
(object sender, NAudio.Wave.WaveInEventArgs args) =>
{
lock (writeLock)
{
if (!writeMore) return;
streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
AudioContent = Google.Protobuf.ByteString
.CopyFrom(args.Buffer, 0, args.BytesRecorded)
}).Wait();
}
};
// Print responses as they arrive.
Task printResponses = Task.Run(async () =>
{
while (await streamingCall.ResponseStream.MoveNext(default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream
.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
Console.WriteLine(alternative.Transcript);
//Textbox1.Text = alternative.Transcript;
}
}
}
});
waveIn.StartRecording();
Console.WriteLine("Speak now.");
Result_Tone.Text = "Speak now:\n\n";
await Task.Delay(TimeSpan.FromSeconds(seconds));
// Stop recording and shut down.
waveIn.StopRecording();
lock (writeLock) writeMore = false;
await streamingCall.WriteCompleteAsync();
await printResponses;
return 0;
}
My problem is that I want to update the content of the Textbox1control but it doesn´t work. It writes perfectly the output into the console with the line Console.WriteLine(alternative.Transcript); but not into my textbox.
If someone could help I would appreciate so much his help.
The problem is that you're using Task.Run, which means your code will be running on a thread-pool thread.
Instead of calling Task.Run(), just move that code into a separate async method:
async Task DisplayResponses(IAsyncEnumerator<StreamingRecognizeResponse> responses)
{
while (await responses.MoveNext(default(CancellationToken)))
{
foreach (var result in responses.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
Textbox1.Text = alternative.Transcript;
}
}
}
}
Then call that method directly (without Task.Run) from code that's already on the UI thread (e.g. an event handler).
The async machinery will make sure that after the await expression, you're back on the UI thread (the same synchronization context). So the assignment to the Text property will occur on the UI thread, and all should be well.
For example:
// This would be registered as the event handler for a button
void HandleButtonClick(object sender, EventArgs e)
{
var stream = client.StreamingRecognize();
// Send the initial config request
await stream.WriteAsync(...);
// Presumably you want to send audio data...
StartSendingAudioData(stream);
await DisplayResponses(stream.ResponseStream);
}
Tasks run on seperate threads, so you must Invoke an action that will be performed on the control's thread
Textbox1.Invoke(new Action(() =>
{
Textbox1.Text= "";
}));
Edit: For WPF, I believe the equivalent is
Textbox1.Dispatcher.Invoke(new Action(() =>
{
Textbox1.Text= "";
}));
have you tried using Dispatcher.InvokeASync()?
await Dispatcher.InvokeAsync(() => {while (await streamingCall.ResponseStream.MoveNext(default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream
.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
Textbox1.Text = alternative.Transcript;
}
}
}});

Messages are not getting delayed in Botframework

I'm posting two messages back to the user as a reply as below,
static Timer t = new Timer(new TimerCallback(TimerEvent));
static Timer t1 = new Timer(new TimerCallback(TimerEventInActivity));
static int timeOut = Convert.ToInt32(ConfigurationManager.AppSettings["disableEndConversationTimer"]); //3600000
public static void CallTimer(int due) {
t.Change(due, Timeout.Infinite);
}
public static void CallTimerInActivity(int due) {
t1.Change(due, Timeout.Infinite);
}
public async static Task PostAsyncWithDelay(this IDialogContext ob, string text) {
try {
var message = ob.MakeMessage();
message.Type = Microsoft.Bot.Connector.ActivityTypes.Message;
message.Text = text;
await PostAsyncWithDelay(ob, message);
CallTimer(300000);
if ("true".Equals(ConfigurationManager.AppSettings["disableEndConversation"])) {
CallTimerInActivity(timeOut);
}
} catch (Exception ex) {
Trace.TraceInformation(ex.Message);
}
}
await context.PostAsyncWithDelay("Great!");
await context.PostAsyncWithDelay("I can help you with that.");
But, there is no delay between them when received. Both messages are received in one go.
How can I delay the second message with some time?
In Root Dialog
To delay your message you can use Task.Delay method. Change your PostAsyncWithDelay as:
public async static Task PostAsyncWithDelay(IDialogContext context, string text)
{
await Task.Delay(4000).ContinueWith(t =>
{
var message = context.MakeMessage();
message.Text = text;
using (var scope = DialogModule.BeginLifetimeScope(Conversation.Container, message))
{
var client = scope.Resolve<IConnectorClient>();
client.Conversations.ReplyToActivityAsync((Activity)message);
}
});
}
You can call PostAsyncWithDelay method when you want to delay a message, otherwise use context.PostAsync method to send your messages.
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
//Sending a message nomally
await context.PostAsync("Hi");
//Notify the user that the bot is typing
var typing = context.MakeMessage();
typing.Type = ActivityTypes.Typing;
await context.PostAsync(typing);
//The message you want to delay.
//NOTE: Do not use context.PostAsyncWithDelay instead simply call the method.
await PostAsyncWithDelay(context, "2nd Hi");
}
OUTPUT
How can I delay the second message with some time?
If you’d like to delay sending the second message, you can try the following code snippet:
await context.PostAsync($"You sent {activity.Text} at {DateTime.Now}");
Task.Delay(5000).ContinueWith(t =>
{
using (var scope = Microsoft.Bot.Builder.Dialogs.Internals.DialogModule.BeginLifetimeScope(Conversation.Container, activity))
{
var client = scope.Resolve<IConnectorClient>();
Activity reply = activity.CreateReply($"I can help you with that..");
client.Conversations.ReplyToActivityAsync(reply);
}
});
context.Wait(MessageReceivedAsync);
Besides, as others mentioned in comments, the method PostAsyncWithDelay seems not a built-in method in Bot Builder SDK. If you try to achieve the requirement and defined that custom method, you can post the code of that method.
To make all replies delay, you may insert this directly in the controller.
if (activity.Type == ActivityTypes.Message)
{
var connector = new ConnectorClient(new Uri(activity.ServiceUrl));
Activity isTypingReply = activity.CreateReply();
isTypingReply.Type = ActivityTypes.Typing;
await connector.Conversations.ReplyToActivityAsync(isTypingReply);
var message = isTypingReply;
await Task.Delay(4000).ContinueWith(t =>
{
using (var scope = DialogModule.BeginLifetimeScope(Conversation.Container, message))
{
}
});
await Conversation.SendAsync(activity, () => new Dialogs.RootDialog());
}

BrokeredMessage disposed after accessing from different thread

This might be a duplicate of this question but that's confused with talk about batching database updates and still has no proper answer.
In a simple example using Azure Service Bus queues, I can't access a BrokeredMessage after it's been placed on a queue; it's always disposed if I read the queue from another thread.
Sample code:
class Program {
private static string _serviceBusConnectionString = "XXX";
private static BlockingCollection<BrokeredMessage> _incomingMessages = new BlockingCollection<BrokeredMessage>();
private static CancellationTokenSource _cancelToken = new CancellationTokenSource();
private static QueueClient _client;
static void Main(string[] args) {
// Set up a few listeners on different threads
Task.Run(async () => {
while (!_cancelToken.IsCancellationRequested) {
var msg = _incomingMessages.Take(_cancelToken.Token);
if (msg != null) {
try {
await msg.CompleteAsync();
Console.WriteLine($"Completed Message Id: {msg.MessageId}");
} catch (ObjectDisposedException) {
Console.WriteLine("Message was disposed!?");
}
}
}
});
// Now set up our service bus reader
_client = GetQueueClient("test");
_client.OnMessageAsync(async (message) => {
await Task.Run(() => _incomingMessages.Add(message));
},
new OnMessageOptions() {
AutoComplete = false
});
// Now start sending
Task.Run(async () => {
int sent = 0;
while (!_cancelToken.IsCancellationRequested) {
var msg = new BrokeredMessage();
await _client.SendAsync(msg);
Console.WriteLine($"Sent {++sent}");
await Task.Delay(1000);
}
});
Console.ReadKey();
_cancelToken.Cancel();
}
private static QueueClient GetQueueClient(string queueName) {
var namespaceManager = NamespaceManager.CreateFromConnectionString(_serviceBusConnectionString);
if (!namespaceManager.QueueExists(queueName)) {
var settings = new QueueDescription(queueName);
settings.MaxDeliveryCount = 10;
settings.LockDuration = TimeSpan.FromSeconds(5);
settings.EnableExpress = true;
settings.EnablePartitioning = true;
namespaceManager.CreateQueue(settings);
}
var factory = MessagingFactory.CreateFromConnectionString(_serviceBusConnectionString);
factory.RetryPolicy = new RetryExponential(minBackoff: TimeSpan.FromSeconds(0.1), maxBackoff: TimeSpan.FromSeconds(30), maxRetryCount: 100);
var queueClient = factory.CreateQueueClient(queueName);
return queueClient;
}
}
I've tried playing around with settings but can't get this to work. Any ideas?
Answering my own question with response from Serkant Karaca # Microsoft here:
Very basic rule and I am not sure if this is documented. The received message needs to be processed in the callback function's life time. In your case, messages will be disposed when async callback completes, this is why your complete attempts are failing with ObjectDisposedException in another thread.
I don't really see how queuing messages for further processing helps on the throughput. This will add more burden to client for sure. Try processing the message in the async callback, that should be performant enough.
Bugger.

How can I make many pings asynchronously at the same time?

I just can't seem to understand how to structure an asynchronous call to SendPingAsync. I want to loop through a list of IP addresses and ping them all asynchronously before moving on in the program... right now it takes forever to go through all of them one at a time. I asked a question about it earlier thinking I'd be able to figure out async but apparently I was wrong.
private void button1_Click(object sender, EventArgs e)
{
this.PingLoop();
MessageBox.Show("hi"); //for testing
}
public async void PingLoop()
{
Task<int> longRunningTask = PingAsync();
int result = await longRunningTask;
MessageBox.Show("async call is finished!");
//eventually want to loop here but for now just want to understand how this works
}
private async Task<int> PingAsync()
{
Ping pingSender = new Ping();
string reply = pingSender.SendPingAsync("www.google.com", 2000).ToString();
pingReplies.Add(reply); //what should i be awaiting here??
return 1;
}
I'm afraid I just don't get what is really going on here enough... when should I return a task? When I run this as is I just get a frozen UI and a ping error. I have read the MSDN documentation and tons of questions here and I'm just not getting it.
You'd want to do something like:
private async Task<List<PingReply>> PingAsync()
{
var tasks = theListOfIPs.Select(ip => new Ping().SendPingAsync(ip, 2000));
var results = await Task.WhenAll(tasks);
return results.ToList();
}
This will start off one request per IP in theListOfIPs asynchronously, then asynchronously wait for them all to complete. It will then return the list of replies.
Note that it's almost always better to return the results vs. setting them in a field, as well. The latter can lead to bugs if you go to use the field (pingReplies) before the asynchronous operation completes - by returning, and adding the range to your collection after the call is made with await, you make the code more clear and less bug prone.
What you do here pingSender.SendPingAsync("www.google.com", 2000).ToString(); doesn't make much sense.
Instead you should return pingSender.SendPingAsync("www.google.com", 2000) and
await Task.WhenAll(your all ping requests)
What you want is to start all pings at once:
var pingTargetHosts = ...; //fill this in
var pingTasks = pingTargetHosts.Select(
host => new Ping().SendPingAsync(host, 2000)).ToList();
Now the pings are running. Collect their results:
var pingResults = await Task.WhenAll(pingTasks);
Now the concurrent phase of the processing is done and you can examine and process the results.
Here is how I do it
private delegate void scanTargetDelegate(IPAddress ipaddress);
private Task<PingReply> pingAsync(IPAddress ipaddress)
{
var tcs = new TaskCompletionSource<PingReply>();
try
{
AutoResetEvent are = new AutoResetEvent(false);
Ping ping = new Ping();
ping.PingCompleted += (obj, sender) =>
{
tcs.SetResult(sender.Reply);
};
ping.SendAsync(ipaddress, new object { });
}
catch (Exception)
{
}
return tcs.Task;
}
in a BackgroundWorker I do this
List<Task<PingReply>> pingTasks = new List<Task<PingReply>>();
addStatus("Scanning Network");
foreach (var ip in range)
{
pingTasks.Add(pingAsync(ip));
}
Task.WaitAll(pingTasks.ToArray());
addStatus("Network Scan Complete");
scanTargetDelegate d = null;
IAsyncResult r = null;
foreach (var pingTask in pingTasks)
{
if (pingTask.Result.Status.Equals(IPStatus.Success))
{
d = new scanTargetDelegate(scanTarget); //do something with the ip
r = d.BeginInvoke(pingTask.Result.Address, null, null);
Interlocked.Increment(ref Global.queuedThreads);
}
else
{
if (!ownIPs.Contains(pingTask.Result.Address))
{
failed.Add(pingTask.Result.Address);
}
}
}
if (r != null)
{
WaitHandle[] waits = new WaitHandle[] { r.AsyncWaitHandle };
WaitHandle.WaitAll(waits);
}
public static async Task<bool> PingAsync(string host)
{
try
{
var ping = new System.Net.NetworkInformation.Ping();
var reply = await ping.SendTaskAsync(host);
return (reply.Status == System.Net.NetworkInformation.IPStatus.Success);
}
catch { return false; }
}

Categories