We Keep Coding

Calling async method without awaiting blocks execution of the rest of ASP.NET Core services

I'm currently working on ASP.NET Core WebApp, which consist of web server and two long-running services– TCP Server (for managing my own clients) and TCP Client (integration with external platform).
Both of services are running alongside web sever– I achieved that, by making them inherit from BackgroundService and injecting to DI in this way:
services.AddHostedService(provider => provider.GetService<TcpClientService>());
services.AddHostedService(provider => provider.GetService<TcpServerService>());
Unfortunately, while development I ran into weird issue (which doesn't let me sleep at night so at this point I beg for your help). For some reason async code in TcpClientService blocks execution of other services (web server and tcp server).
using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ClientService.AsyncPoblem
{
public class TcpClientService : BackgroundService
{
private readonly ILogger<TcpClientService> _logger;
private bool Connected { get; set; }
private TcpClient TcpClient { get; set; }
public TcpClientService(ILogger<TcpClientService> logger)
{
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
if (Connected)
{
await Task.Delay(100, stoppingToken); // check every 100ms if still connected
}
else
{
TcpClient = new TcpClient("localhost", 1234);
HandleClient(TcpClient); // <-- Call causing the issue
_logger.Log(LogLevel.Debug, "After call");
}
}
catch (Exception e)
{
// log the exception, wait for 3s and try again
_logger.Log(LogLevel.Critical, "An error occured while trying to connect with server.");
_logger.Log(LogLevel.Critical, e.ToString());
await Task.Delay(3000, stoppingToken);
}
}
}
private async Task HandleClient(TcpClient client)
{
Connected = true;
await using var ns = client.GetStream();
using var streamReader = new StreamReader(ns);
var msgBuilder = new StringBuilder();
bool reading = false;
var buffer = new char[1024];
while (!streamReader.EndOfStream)
{
var res = await streamReader.ReadAsync(buffer, 0, 1024);
foreach (var value in buffer)
{
if (value == '\x02')
{
msgBuilder.Clear();
reading = true;
}
else if (value == '\x03')
{
reading = false;
if (msgBuilder.Length > 0)
{
Console.WriteLine(msgBuilder);
msgBuilder.Clear();
}
}
else if (value == '\x00')
{
break;
}
else if (reading)
{
msgBuilder.Append(value);
}
}
Array.Clear(buffer, 0, buffer.Length);
}
Connected = false;
}
}
}
Call causing the issue is located in else statement of ExecuteAsync method
else
{
TcpClient = new TcpClient("localhost", 1234);
HandleClient(TcpClient); // <-- Call causing the issue
_logger.Log(LogLevel.Debug, "After call");
}
The code reads properly from the socket, but it blocks initialization of WebServer and TcpServer. Actually, even log method is not being reached. No matter if I put await in front of HandleClient() or not, the code behaves the same.
I've done some tests, and I figured out that this piece of code is not blocking anymore ("After call" log shows up):
else
{
TcpClient = new TcpClient("localhost", 1234);
await Task.Delay(1);
HandleClient(TcpClient); // <- moving Task.Delay into HandleClient also works
_logger.Log(LogLevel.Debug, "After call");
}
This also works like a charm (if I try to await Task.Run(), it will block "After call" log, but rest of app will start with no problem):
else
{
tcpClient = new TcpClient("localhost", 6969);
Connected = true;
Task.Run(() => ReceiveAsync(tcpClient));
_logger.Log(LogLevel.Debug, "After call");
}
There is couple more combinations which make it work, but my question is– why other methods work (especially 1ms delay- this completely shut downs my brain) and firing HandleClient() without await doesn't? I know that fire and forget may not be the most elegant solution, but it should work and do it's job shouldn't it? I searched for almost a month, and still didn't find a single explanation for that. At this point I have hard time falling asleep at night, cause I have no one to ask and can't stop thinking about that..
Update
(Sorry for disappearing for over a day without any answers)
After many many hours of investigation, I started debugging once again. Every time I would hit while loop in HandleClient(), I was losing control over debugger, program seemed to continue to work, but it would never reach await streamReader.ReadAsync(). At some point I decided to change condition in the while loop to true (I have no idea why I didn't think of trying it before), and everything began to work as expected. Messages would get read from tcp socket, and other services would fire up without any issues.
Here is piece of code causing issue
while (!streamReader.EndOfStream) <----- issue
{
var res = await streamReader.ReadAsync(buffer, 0, 1024);
// ...
After that observation, I decided to print out the result of EndOfStream before reaching the loop, to see what happens
Console.WriteLine(streamReader.EndOfStream);
while (!streamReader.EndOfStream)
{
var res = await streamReader.ReadAsync(buffer, 0, 1024);
// ...
Now the exact same thing was happening, but before even reaching the loop!
Explanation
Note:
I'm not senior programmer, especially when it comes to dealing with asynchronous TCP communication so I might be wrong here, but I will try to do my best.
streamReader.EndOfStream is not a regular field, it is a property, and it has logic inside it's getter.
This is how it looks like from the inside:
public bool EndOfStream
{
get
{
ThrowIfDisposed();
CheckAsyncTaskInProgress();
if (_charPos < _charLen)
{
return false;
}
// This may block on pipes!
int numRead = ReadBuffer();
return numRead == 0;
}
}
EndOfStream getter is synchronous method. To detect whether stream has ended or not, it calls ReadBuffer(). Since there is no data in the buffer yet and stream hasn't ended, method hangs until there is some data to read. Unfortunately it cannot be used in asynchronous context, it will always block (unfortunately because it seems to be the only way to instantly detect interrupted connection, broken cable or end of stream).
I don't have finished piece of code yet, I need to rewrite it and add some broken connection detection. I will post my solution I soon as I finish.
I would like to thank everyone for trying to help me, and especially #RoarS. who took biggest part in discussion, and spent some of his own time to take a closer look at my issue.

This is poorly documented behaviour of the BackgroundService class. All registered IHostedService will be started sequentially in the order they were registered. The application will not start until each IHostedService has returned from StartAsync. A BackgroundService is an IHostedService that starts your ExecuteAsync task before returning from StartAsync. Async methods will run until their first call to await an incomplete task before returning.
TLDR; If you don't await anything in your ExecuteAsync method, the server will never start.
Since you aren't awaiting that async method, your code boils down to;
while(true)
HandleClient(...);
(Do you really want to spawn an infinite number of TcpClient as fast as the CPU will go?). There's a really easy fix;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await Task.Yield();
// ...
}

How to correctly close the EventHubReceiver when working with Azure IoT in C#?

I am writing an application that should be able to read and display IoT data. The basic functionality works for me with this code (I removed some checks etc so that the code would the shorter):
public void Run()
{
_eventHubClient = EventHubClient.CreateFromConnectionString(ConnectionString, "messages/events");
var partitions = _eventHubClient.GetRuntimeInformation().PartitionIds;
cts = new CancellationTokenSource();
var tasks = partitions.Select(partition => ReceiveMessagesFromDeviceAsync(partition, cts.Token));
Task.WaitAll(tasks.ToArray());
}
public void Cancel()
{
cts.Cancel();
}
private async Task ReceiveMessagesFromDeviceAsync(string partition, CancellationToken cancellationToken)
{
var eventHubReceiver = _eventHubClient.GetDefaultConsumerGroup().CreateReceiver(partition, DateTime.UtcNow);
while (true)
{
if (cancellationToken.IsCancellationRequested)
{
break;
}
var eventData = await eventHubReceiver.ReceiveAsync(new TimeSpan(0,0,1));
var data = Encoding.UTF8.GetString(eventData.GetBytes());
Console.WriteLine("Message received at {2}. Partition: {0} Data: '{1}'", partition, data, eventData.EnqueuedTimeUtc);
}
}
My problem is that I need to be able to stop and restart the connection again. Everything works okay until the moment when I start it for the 6th time, then I get the "QuotaExceededException": "Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5". I have googled the exception and I understand the problem, what I don't know is how to correctly close the previous receivers after I close a connection, so that I could open it again later. I have tried calling
eventHubReceiver.Close()
in the Cancel() method but it didn't seem to help.
I would be very grateful for any hints on how to solve this, thanks.

C#, Maximize Thread Concurrency

With the help of Google and community, I was able to build a nice set of methods allowing me to asynchronously call a function. This function is testing remote host properties, so it is idling most of the time. For this reason I would like to maximize the number of concurrent threads launched such that all calls can be processed in the minimum amount of time.
Here is the Code I have so far:
// Check remote host connectivity
public static class CheckRemoteHost
{
// Private Class members
private static bool AllDone = false;
private static object lockObj = new object();
private static List<string> IPs;
// Wrapper: manage async method <Ping>
public static List<string> Ping(HashSet<string> IP_Ports, int TimeoutInMS = 100)
{// async worker method: check remote host via <Ping>
// Locals
IPs = new List<string>();
// Perform remote host check
AllDone = false;
Ping_check(IP_Ports, TimeoutInMS);
while (!AllDone) { CommonLib.Utils.ApplicationWait(10, 10); }
// Finish
return IPs;
}
private static async void Ping_check(HashSet<string> IP_Ports, int timeout)
{
// Locals
var tasks = new List<Task>();
// Build task-set for parallel Ping checks
foreach (string host in IP_Ports)
{
var task = PingAndUpdateAsync(host, timeout);
tasks.Add(task);
}
// Start execution queue
await Task.WhenAll(tasks).ContinueWith(t =>
{
AllDone = true;
});
}
private static async Task PingAndUpdateAsync(string ip, int timeout)
{
// Locals
System.Net.NetworkInformation.Ping ping;
System.Net.NetworkInformation.PingReply reply;
try
{
ping = new System.Net.NetworkInformation.Ping();
reply = await ping.SendPingAsync(ip, timeout);
if(reply.Status == System.Net.NetworkInformation.IPStatus.Success)
{
lock (lockObj)
{
IPs.Add(ip);
}
}
}
catch
{
// do nothing
}
}
}// end public static class CheckRemoteHost
This code is tested quite extensively, and the code seems stable and reliably report live hosts. Having said that, I know that it only spawns 8 threads at a time (= number of logical core on my test machine).
The key portion of the code is this:
// Start execution queue
await Task.WhenAll(tasks).ContinueWith(t =>
{
AllDone = true;
});
This is where I would like to increase/ maximize the number of concurrently launched threads to something like 25 per core (remember the thread job is 99% idle).
So far, my thread concurrency research has brought up the explicit thread and Parallel.For approaches. However, these seem to have the same shortcoming of spawning no more than 8 threads.
Any help would be very much appreciated, so thank you very much in advance everyone for looking!

You're making your life hard with the code you have. It's got a lot of plumbing that isn't needed and you're sharing static fields that would cause your code to fail if you called Ping a second time while the first one is running.
You need to get rid of all of that stuff.
I'd suggest using Microsoft's Reactive Framework - just NuGet "System.Reactive" and add using System.Reactive.Linq; to your code. Then you can do this:
public static class CheckRemoteHost
{
public static IList<string> Ping(HashSet<string> IP_Ports, int TimeoutInMS = 100)
{
var query =
from host in IP_Ports.ToObservable()
from status in Observable.FromAsync(() => PingAsync(host, TimeoutInMS))
where status
select host;
return query.ToList().Wait();
}
private static async Task<bool> PingAsync(string ip, int timeout)
{
try
{
var ping = new System.Net.NetworkInformation.Ping();
var reply = await ping.SendPingAsync(ip, timeout);
return reply.Status == System.Net.NetworkInformation.IPStatus.Success;
}
catch
{
return false;
}
}
}
That's it. That's all of the code you need. It's automatically maximising the thread use to get the job done.

c# data not processing correctly in multithreading

I have a product dropdown and selecting the product connects to a websocket and get the feed messages for that product. Once the (1) feed messages starts coming I have to then (2) get the order book and then (3) process the feed messages. So the first and last task would run asynchronously. For this I have written the following code:
void OnReceivingFeedMessage()
{
concurrentQueue.Enqueue(message);
if (!messageStreamStarted) // only first time get order book
{
messageStreamStarted = true;
GetOrderBookData();
}
}
private void GetOrderBookData()
{
MarketData m = new MarketData();
ProductOrderBook p = m.GetProductOrderBook(productId);
bidsList = p.bids;
asksList = p.asks;
isOrderBookUpdated = true;
Task task3 = Task.Run(() => KickStartToProcessQueue());
}
private void KickStartToProcessQueue()
{
while (threadProcessQueueExist)
{
int recordCountNew = concurrentQueue.Count();
if (recordCountNew != 0)
{
if (isOrderBookUpdated)
{
ProcessQueueMessages();
}
}
}
}
private void ProcessQueueMessages()
{
if (!concurrentQueue.IsEmpty)
{
string jsonString;
while (concurrentQueue.TryDequeue(out jsonString))
{
// have to insert the record in existing order book
}
}
}
This works perfectly for the first time. But when I change the product and reconnect the things mess up and the data is not processed properly. The code written on product selectedindex change
private void CloseAndReconnectToGetWebsocketFeed()
{
w.CloseWebsocketConnection();
messageStreamStarted = false;
isOrderBookUpdated = false;
ConcurrentQueue<string> wssMessagesQueue = new ConcurrentQueue<string>();
concurrentQueue = wssMessagesQueue;
ConnectAndGetWebsocketFeedMessages(); // this calls OnReceivingFeedMessage
}
I am new to multi-threading so not sure if I need to use lock or async/await or something else. What am I doing wrong in the above code?
It is running fine when run first time but the moment I change the product and do the same processing again it starts giving problems. Can someone please advise how can I clear all the resources before doing the same steps again and again

I think you are writing unneccesarily complicated code. I'm not 100 % sure, what your problem is, but here are some things that might help you.
Use a BlockingCollection<T>
With that class, you can stop your consumer-thread until new messages are coming in. Here's a simple example on how these things are working:
BlockingCollection<string> collection = new BlockingCollection<string>(new ConcurrentQueue<string>());
Task t = Task.Run(() =>
{
while (collection.TryTake(out string item, Timeout.Infinite))
{
Console.WriteLine($"Started reading {item}...");
Thread.Sleep(1000); //simulate intense work
Console.WriteLine($"Done reading {item}");
}
});
while (true)
{
//This could be your OnReceivingFeedMessage()
string input = Console.ReadLine();
if (input == "stop")
{
Console.WriteLine("Stopping...");
collection.CompleteAdding();
break;
}
else
{
collection.Add(input);
}
}
t.Wait();
The task t will wait until there are items in collection. When items are "received" (here simply via console input), they are added to your list.
Dispatch new tasks to work on the input
Pretty simple:
while (true)
{
string item = Console.ReadLine();
Task.Run(() =>
{
Console.WriteLine($"Started reading {item}...");
Thread.Sleep(1000); //simulate intense work
Console.WriteLine($"Done reading {item}");
});
}
This also has the advantage (or disadvantage) that the tasks are running all in parallel. That means that you can't rely on the order they are worked on, but they will process much faster.
By the way: Both of these approaches have the advantage that you don't have busy waiting. From your question:
while (threadProcessQueueExist)
{
int recordCountNew = concurrentQueue.Count();
if (recordCountNew != 0)
{
if (isOrderBookUpdated)
{
ProcessQueueMessages();
}
}
}
This code will create busy waiting as long as nothing is in your queue, meaning that one core of your processor will be at very high load without actually doing anything. It is considered to be bad practice.

WP8.1 - CancellationToken and async still await forever and never end

I have a really strange problem and I don't know how to solve it.
I have these two methods in different classes.
The first one is triggered when a button in the CommandBar is pressed.
EDIT: I created two similar but smaller methods to show you the problem:
private async void runCode(object sender, RoutedEventArgs e)
{
BottomAppBar.IsEnabled = false;
object result = await endlessLoopTest();
BottomAppBar.IsEnabled = true;
}
private async Task<object> endlessLoopTest()
{
var tokenSource = new System.Threading.CancellationTokenSource(500);
try
{
await Task.Run(() =>
{
while (true)
{
//Infinite loop to test the code
}
}, tokenSource.Token);
return null;
}
catch (OperationCanceledException)
{
return new TextBlock();
}
}
I added a cancellationToken that expires after 1500ms (I assume that if the interpreter takes longer to process the code, it has been trapped in a loop).
The first time I try this it usually works, but if I try again, the CommandBar buttons never get enabled again, so I assume that task is being awaited forever, and I don't know why, as I added that cancellationToken.
Do you know what could be wrong here?
Thanks for your help!
Sergio

You are about 2/3's of the way there. When using a CancellationToken + CancellationTokenSournce, one must ask the token if it was cancelled. There are a number of ways to subscribe to that, including calling the token's ThrowIfCancelledRequest method or checking the token's Boolean property IsCancellationRequested and breaking out of the loop. See Cancellation in Managed Threads.
Here's a small example that can run in a Console app. Note, in UI based apps, use await, not Task.Wait().
private static void CancelTask()
{
CancellationTokenSource cts = new CancellationTokenSource(750);
Task.Run(() =>
{
int count = 0;
while (true)
{
Thread.Sleep(250);
Console.WriteLine(count++);
if (cts.Token.IsCancellationRequested)
{
break;
}
}
}, cts.Token).Wait();
}
The result is 0 1 2 and then the Task and program exit.