We Keep Coding

How can I send a batch of messages to an Azure EventHub? (C# WinForm and .NET 5.0)

I am building a telemetry simulator to send messages to an Azure EventHub (WinForm, .NET 5.0).
I use backgroundworkers for each telemetry device, the code below is the DoWork method of one of the devices. The app works fine when I output the messages to the console.
The problem occurs when I add the (commented out) EventHub code shown in the while loop below. There are two issues:
Backgroundworker.ReportProgress fails with: System.InvalidOperationException: 'This operation has already had OperationCompleted called on it and further calls are illegal.'
I also can no longer cancel the process (i.e. truckWorker.CancellationPending is always false)
If I uncomment the EventHub code, and comment out truckWorker.ReportProgress(i++), the app works and sends messages to the EventHub. Problem is I still can't cancel the operation and of course I loose my progress indicator.
private async void SendTruckMsg(object sender, DoWorkEventArgs e)
{
EventHubProducerClient evClient = (EventHubProducerClient)e.Argument;
List<TruckTelemetry> collection = virtualThingsController.GetTruckTelemetry();
int interval = virtualThingsController.GetTruckTelemetryInterval();
int i = 0; // count messages sent
while (!truckWorker.CancellationPending) // ===> can't cancel operation anymore
{
//using var eventBatch = await evClient.CreateBatchAsync(); // create batch
foreach (TruckTelemetry truckTelemetry in collection)
{
truckTelemetry.Timestamp = DateTime.Now;
string output = JsonConvert.SerializeObject(truckTelemetry);
//eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(output))); // add to batch
}
//await evClient.SendAsync(eventBatch); // send the batch
truckWorker.ReportProgress(i++);
Thread.Sleep(interval);
}
}
Sending messages could be synchronous but there is no simple 'send' method for the EventHubProducerClient.
Appreciate any help, thanks.

Before I touch on the background processing, there's a couple of things that I'm seeing in your snippet that look like they're likely to be a problem in your application:
It appears that you're using a List<TruckTelemetry> on multiple threads, one that adds items and the background operation that publishes them to Event Hubs. The list isn't thread-safe. I'd recommend moving to a ConcurrentQueue<T>, which is thread-safe and will help preserve the ordering of your telemetry.
Your snippet is ignoring the return of the TryAdd method of the EventDataBatch and assuming that every telemetry item in your loop was successfully added to the batch. This can lead to data loss, as you would be unaware if the batch was full and events were not able to be added.
Your snippet doesn't handle the corner case where a telemetry item is too large and could never be published to Event Hubs. Granted, if your telemetry items are tiny, this isn't a case that you're likely to encounter, but I'd recommend including it just to be safe.
With asynchronous operations, the typical approach to working in the background is to start a new Task that you can cancel or wait to complete when it makes sense for the application.
If I translate your general scenario, turning it into a task and addressing my feedback, your telemetry processing would look something like:
public static Task ProcessTruckTelemetry(EventHubProducerClient producer,
VirtualThingsController controller,
CancellationToken cancellationToken,
Action<int> progressCallback) =>
Task.Run(async () =>
{
var eventBatch = default(EventDataBatch);
var totalEventCount = 0;
var interval = controller.GetTruckTelemetryInterval();
// I'm assuming a change to ConcurrentQueue<T> for the
// the telemetry interactions via the controller.
var telemetryQueue = controller.GetTruckTelemetry();
while (!cancellationToken.IsCancellationRequested)
{
// In this example, we'll pump the telemetry queue as long
// long as there are any items in there. If you wanted to
// limit, you could also include a maximum count and stop there.
while ((!cancellationToken.IsCancellationRequested)
&& (telemetryQueue.TryPeek(out var telemetry))
{
// Create a batch if we don't currently have one.
eventBatch ??= (await producer.CreateBatchAsync().ConfigureAwait(false));
// Translate the telemetry data.
telemetry.Timestamp = DateTime.UtcNow;
var serializedTelemetry = JsonSerializer.Serialize(telemetry);
var eventData = new EventData(new BinaryData(serializedTelemetry));
// Attempt to add the event to the batch. If the batch is full,
// send it and clear state so that we know to create a new one.
if (!eventBatch.TryAdd(eventData))
{
// If there are no events in the batch, this event is
// too large to ever publish. We can't recover.
//
// An important note in this scenario is that we have
// already removed the telemetry from the queue. If we don't
// want to lose that, we should take action before throwing the
// exception to capture it.
if (eventBatch.Count == 0)
{
throw new Exception("There was an event too large to publish.");
}
await producer.SendAsync(eventBatch).ConfigureAwait(false);
totalEventCount += eventBatch.Count;
eventBatch.Dispose();
eventBatch = default;
}
}
else
{
telemetryQueue.TryDequeue(out _);
}
// Once we hit this point, there were no telemetry items left
// in the queue. Send any that are held in the event batch.
if ((eventBatch != default) && (eventBatch.Count > 0))
{
await producer.SendAsync(eventBatch).ConfigureAwait(false);
totalEventCount += eventBatch.Count;
eventBatch.Dispose();
eventBatch = default;
}
// Invoke the progress callback with the total count.
progressCallback(totalEventCount);
// Pause for the requested delay before attempting to
// pump the telemetry queue again.
try
{
await Task.Delay(interval, cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Thrown if cancellation is requested while we're in the
// delay. This example is assuming that it isn't interesting
// to the application and swallows it.
}
}
}, cancellationToken);
Which your application would interact with using something similar to:
// This is your master shutdown signal. When you request
// cancellation on this token, all of your background tasks
// should terminate.
using var cancellationSource = new CancellationTokenSource();
var producer = GetProducerClient();
var virtualThingsController = new VirtualThingsController();
try
{
// Define a simple process callback.
Action<int> progressCallback = totalCount =>
Debug.WriteLine($"There have been a total of { totalCount } items published.");
// Start the background processing and capture the tasks. Once the
// call is made, telemetry is being processed in the backgorund until
// the cancellation token is signaled.
var backgroundTasks = new List<Task>();
backgroundTasks.Add
(
ProcessTruckTelemetry(
producer,
virtualThingsController,
cancellationSource.Token,
progressCallback)
);
backgroundTasks.Add
(
ProcessOtherTelemetry(
producer,
virtualThingsController,
cancellationSource.Token,
progressCallback)
);
// The application can do whatever it normally does now.
//
// << STUFF >>
//
// When the application is ready to stop, signal the cancellation
// token and wait for the tasks. We're not calling ConfigureAwait(false)
// here because your application is WinForms, which has some
// sensitivity to the syncrhonization context.
cancellationSource.Cancel();
await Task.WhenAll(backgroundTasks);
}
finally
{
await producer.CloseAsync();
}

Multiple connections with TcpClient, second connection always hangs/does nothing

So I have a TcpClient in a console app that is listening on port 9096. I want the client to be able to handle multiple connections (simultaneous or not). I also do not want to use Threads. I want to use async/await. I also need to be able to gracefully close the app during certain events, being careful not to lose any data. So I need a cancellation token. I have the code mostly working but there are two issues.
First, when the app starts listening and I send it data; everything works correctly as long as the sender is using the same initial connection to the app. Once a new connection (or socket I guess? not clear on the terminology) is established the app does not process the new data.
Second, when the terminate signal is given to the app and the token is canceled the app does not close. I am not getting any exceptions and I cannot figure out what I an doing wrong.
I have looked all over and cannot find an example of a TcpClient that uses async/await with a cancellation token. I also cannot find an example that I have been able to get working that correctly processes multiple connections, without using Threads or other complicated designs. I want the design as simple as possible with as little code as possible while still meeting my requirements. If using threads is the only way to do it I will, but I am soo close to getting it right I feel like I am just missing a little thing.
I am trying to figure this out at my wits end and have exhausted all my ideas.
EDIT: I moved the AcceptTcpClientAsync into the loop as suggested below and it did not change anything. The app functions the same as before.
Program.cs
class Program
{
private static List<Task> _listeners = new List<Task>();
private static readonly CancellationTokenSource cancelSource = new CancellationTokenSource();
static void Main(string[] args)
{
Console.TreatControlCAsInput = false;
Console.CancelKeyPress += (o, e) => {
Console.WriteLine("Shutting down.");
cancelSource.Cancel();
};
Console.WriteLine("Started, press ctrl + c to terminate.");
_listeners.Add(Listen(cancelSource.Token));
cancelSource.Token.WaitHandle.WaitOne();
Task.WaitAll(_listeners.ToArray(), cancelSource.Token);
}
}
Listen
public async Task Listen(CancellationToken token){
var listener = new TcpListener(IPAddress.Parse("0.0.0.0"), 9096);
listener.Start();
Console.WriteLine("Listening on port 9096");
while (!token.IsCancellationRequested) {
// Also tried putting AcceptTcpClientAsync here.
await Task.Run(async () => {
var client = await listener.AcceptTcpClientAsync();
using (var stream = client.GetStream())
using (var streamReader = new StreamReader(stream, Encoding.UTF8))
using (var streamWriter = new StreamWriter(stream, Encoding.UTF8)) {
while (!token.IsCancellationRequested) {
// DO WORK WITH DATA RECEIVED
vat data = await streamReader.ReadAsync();
await streamWriter.WriteLineAsync("Request received.");
}
}
});
}
Console.WriteLine("Stopped Accepting Requests.");
listener.Server.Close();
listener.Stop();
}

This is actually working the way you designed it, however you have only built to receive one connection. I am not going to write a full socket implementation for you (as this can get fairly in-depth). However, as for your main problem, you need to put the AcceptTcpClientAsync in the loop otherwise you won't get any more connections:
var cancellation = new CancellationTokenSource();
...
var listener = new TcpListener(...);
listener.Start();
try
{
while (!token.IsCancellationRequested)
{
var client = await listener.AcceptTcpClientAsync()
...
}
}
finally
{
listener.Stop();
}
// somewhere in another thread
cancellation.Cancel();
Update
I tried that and no behavior changed. Still does not pick up any
connection after the first.
await ...
while (!token.IsCancellationRequested) {
// DO WORK WITH DATA RECEIVED
It's obvious that AcceptTcpClientAsync will never get called again because you are awaiting the task. This method is what accepts the client, if you can't call it, you don't get any more clients.
You cannot block here, which is what you are doing. Please see some socket server examples to get a better idea of how to write a listener.

.NET HttpClient.PostAsync() slow after 3 requests

I am using the .NET 4.5 HttpClient class to make a POST request to a server a number of times. The first 3 calls run quickly, but the fourth time a call to await client.PostAsync(...) is made, it hangs for several seconds before returning the expected response.
using (HttpClient client = new HttpClient())
{
// Prepare query
StringBuilder queryBuilder = new StringBuilder();
queryBuilder.Append("?arg=value");
// Send query
using (var result = await client.PostAsync(BaseUrl + queryBuilder.ToString(),
new StreamContent(streamData)))
{
Stream stream = await result.Content.ReadAsStreamAsync();
return new MyResult(stream);
}
}
The server code is shown below:
HttpListener listener;
void Run()
{
listener.Start();
ThreadPool.QueueUserWorkItem((o) =>
{
while (listener.IsListening)
{
ThreadPool.QueueUserWorkItem((c) =>
{
var context = c as HttpListenerContext;
try
{
// Handle request
}
finally
{
// Always close the stream
context.Response.OutputStream.Close();
}
}, listener.GetContext());
}
});
}
Inserting a debug statement at // Handle request shows that the server code doesn't seem to receive the request as soon as it is sent.
I have already investigated whether it could be a problem with the client not closing the response, meaning that the number of connections the ServicePoint provider allows could be reached. However, I have tried increasing ServicePointManager.MaxServicePoints but this has no effect at all.
I also found this similar question:
.NET HttpClient hangs after several requests (unless Fiddler is active)
I don't believe this is the problem with my code - even changing my code to exactly what is given there didn't fix the problem.

The problem was that there were too many Task instances scheduled to run.
Changing some of the Task.Factory.StartNew calls in my program for tasks which ran for a long time to use the TaskCreationOptions.LongRunning option fixed this. It appears that the task scheduler was waiting for other tasks to finish before it scheduled the request to the server.

Advice on processing giant text file and processing URL's

I'm currently trying to loop through a text file that is about 1.5gb's in size and then use the URL's that are grabbed from it to pull down the html from the site.
For speed I'm trying to process all the HTTP request on a new thread but since C# is not my strongest language but a requirement for what I'm doing I'm a bit confused on good thread practice.
This is how I'm processing the list
private static void Main()
{
const Int32 BufferSize = 128;
using (var fileStream = File.OpenRead("dump.txt"))
using (var streamReader = new StreamReader(fileStream, Encoding.UTF8, true, BufferSize))
{
String line;
var progress = 0;
while ((line = streamReader.ReadLine()) != null)
{
var stuff = line.Split('|');
getHTML(stuff[3]);
progress += 1;
Console.WriteLine(progress);
}
}
}
And I'm pulling down the HTML as so
private static void getHTML(String url)
{
new Thread(() =>
{
var client = new DecompressGzipResponse();
var html = client.DownloadString(url);
}).Start();
}
Though the speeds are fast doing this initially, after about 20 thousand they slow down and eventually after 32 thousand the application will hang and crash. I was under the impression C# threads terminated when the function completed?
Can anyone give any examples/ suggestions on how to do this better?

One very reliable way to do this is by using the producer-consumer pattern. You create a thread-safe queue of URLs (for example, BlockingCollection<Uri>). Your main thread is the producer, which adds items to the queue. You then have multiple consumer threads, each of which reads Urls from the queue and does the HTTP requests. See BlockingCollection.
Setting it up isn't terribly difficult:
BlockingCollection<Uri> UrlQueue = new BlockingCollection<Uri>();
// Main thread starts the consumer threads
Task t1 = Task.Factory.StartNew(() => ProcessUrls, TaskCreationOptions.LongRunning);
Task t2 = Task.Factory.StartNew(() => ProcessUrls, TaskCreationOptions.LongRunning);
// create more tasks if you think necessary.
// Now read your file
foreach (var line in File.ReadLines(inputFileName))
{
var theUri = ExtractUriFromLine(line);
UrlQueue.Add(theUri);
}
// when done adding lines to the queue, mark the queue as complete
UrlQueue.CompleteAdding();
// now wait for the tasks to complete.
t1.Wait();
t2.Wait();
// You could also use Task.WaitAll if you have an array of tasks
The individual threads process the urls with this method:
void ProcessUrls()
{
foreach (var uri in UrlQueue.GetConsumingEnumerable())
{
// code here to do a web request on that url
}
}
That's a simple and reliable way to do things, but it's not especially quick. You can do much better by using a second queue of WebCient objects that make asynchronous requests For example, say you want to have 15 asynchronous requests. You start the same way with a BlockingCollection, but you only have one persistent consumer thread.
const int MaxRequests = 15;
BlockingCollection<WebClient> Clients = new BlockingCollection<WebClient>();
// start a single consumer thread
var ProcessingThread = Task.Factory.StartNew(() => ProcessUrls, TaskCreationOptions.LongRunning);
// Create the WebClient objects and add them to the queue
for (var i = 0; i < MaxRequests; ++i)
{
var client = new WebClient();
// Add an event handler for the DownloadDataCompleted event
client.DownloadDataCompleted += DownloadDataCompletedHandler;
// And add this client to the queue
Clients.Add(client);
}
// add the code from above that reads the file and populates the queue
Your processing function is somewhat different:
void ProcessUrls()
{
foreach (var uri in UrlQueue.GetConsumingEnumerable())
{
// Wait for an available client
var client = Clients.Take();
// and make an asynchronous request
client.DownloadDataAsync(uri, client);
}
// When the queue is empty, you need to wait for all of the
// clients to complete their requests.
// You know they're all done when you dequeue all of them.
for (int i = 0; i < MaxRequests; ++i)
{
var client = Clients.Take();
client.Dispose();
}
}
Your DownloadDataCompleted event handler does something with the data that was downloaded, and then adds the WebClient instance back to the queue of clients.
void DownloadDataCompleteHandler(Object sender, DownloadDataCompletedEventArgs e)
{
// The data downloaded is in e.Result
// be sure to check the e.Error and e.Cancelled values to determine if an error occurred
// do something with the data
// And then add the client back to the queue
WebClient client = (WebClient)e.UserState;
Clients.Add(client);
}
This should keep you going with 15 concurrent requests, which is about all you can do without getting a bit more complicated. Your system can likely handle many more concurrent requests, but the way that WebClient starts asynchronous requests requires some synchronous work up front, and that overhead makes 15 about the maximum number you can handle.
You might be able to have multiple threads initiating the asynchronous requests. In that case, you could potentially have as many threads as you have processor cores. So on a quad core machine, you could have the main thread and three consumer threads. With three consumer threads this technique could give you 45 concurrent requests. I'm not certain that it scales that well, but it might be worth a try.
There are ways to have hundreds of concurrent requests, but they're quite a bit more complicated to implement.

You need thread management.
My advice is to use Tasks instead of creating your own Threads.
By using the Task Parallel Library, you let the runtime deal with the thread management. By default, it will allocate your tasks on threads from the ThreadPool, and will allow a level of concurrency which is contingent on the number of CPU cores you have. It will also reuse existing Threads when they become available instead of wasting time creating new ones.
If you want to get more advanced, you can create your own task scheduler to manage the scheduling aspect yourself.
See also What is difference between Task and Thread?

Processing Messages in Parallel in Azure Service Bus

Problem: I've got tons of emails to send, presently, an average of 10 emails in the queue at any point in time. The code I have process the queue one at a time; that is, receive the message, process it and eventually send the email. This cause a considerably delay in sending emails to users when they signup for the service.
I've begun to think of modifying the code to process the messages in parrallel say 5 asynchronously. I'm imagining writing a method and using the CTP to call this method in parallel, say, 5 times.
I'm a little bit lost in how to implement this. The cost of making a mistake is exceedingly great as users will get disappointed if things go wrong.
Request: I need help in writing code that process messages in Azure service bus in parallel.
Thanks.
My code in a nutshell.
Public .. Run()
{
_myQueueClient.BeginReceive(ProcessUrgentEmails, _myQueueClient);
}
void ProcessUrgentEmails(IAsyncResult result)
{
//casted the `result` as a QueueClient
//Used EndReceive on an object of BrokeredMessage
//I processed the message, then called
sendEmail.BeginComplete(ProcessEndComplete, sendEmail);
}
//This method is never called despite having it as callback function above.
void ProcessEndComplete(IAsyncResult result)
{
Trace.WriteLine("ENTERED ProcessEndComplete method...");
var bm = result.AsyncState as BrokeredMessage;
bm.EndComplete(result);
}

This page gives you performance tips when using Windows Azure Service Bus.
About parallel processing, you could have a pool of threads for processing, and every time you get a message, you just grab one of that pool and assign it a message. You need to manage that pool.
OR, you could retrieve multiple messages at once and process them using TPL... for example, the method BeginReceiveBatch/EndReceiveBatch allows you to retrieve multiple "items" from Queue (Async) and then use "AsParallel" to convert the IEnumerable returned by the previous methods and process the messages in multiple threads.
VERY simple and BARE BONES sample:
var messages = await Task.Factory.FromAsync<IEnumerable<BrokeredMessage>>(Client.BeginReceiveBatch(3, null, null), Client.EndReceiveBatch);
messages.AsParallel().WithDegreeOfParallelism(3).ForAll(item =>
{
ProcessMessage(item);
});
That code retrieves 3 messages from queue and processes then in "3 threads" (Note: it is not guaranteed that it will use 3 threads, .NET will analyze the system resources and it will use up to 3 threads if necessary)
You could also remove the "WithDegreeOfParallelism" part and .NET will use whatever threads it needs.
At the end of the day there are multiple ways to do it, you have to decide which one works better for you.
UPDATE: Sample without using ASYNC/AWAIT
This is a basic (without error checking) sample using regular Begin/End Async pattern.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
namespace WorkerRoleWithSBQueue1
{
public class WorkerRole : RoleEntryPoint
{
// The name of your queue
const string QueueName = "QUEUE_NAME";
const int MaxThreads = 3;
// QueueClient is thread-safe. Recommended that you cache
// rather than recreating it on every request
QueueClient Client;
bool IsStopped;
int dequeueRequests = 0;
public override void Run()
{
while (!IsStopped)
{
// Increment Request Counter
Interlocked.Increment(ref dequeueRequests);
Trace.WriteLine(dequeueRequests + " request(s) in progress");
Client.BeginReceive(new TimeSpan(0, 0, 10), ProcessUrgentEmails, Client);
// If we have made too many requests, wait for them to finish before requesting again.
while (dequeueRequests >= MaxThreads && !IsStopped)
{
System.Diagnostics.Trace.WriteLine(dequeueRequests + " requests in progress, waiting before requesting more work");
Thread.Sleep(2000);
}
}
}
void ProcessUrgentEmails(IAsyncResult result)
{
var qc = result.AsyncState as QueueClient;
var sendEmail = qc.EndReceive(result);
// We have received a message or has timeout... either way we decrease our counter
Interlocked.Decrement(ref dequeueRequests);
// If we have a message, process it
if (sendEmail != null)
{
var r = new Random();
// Process the message
Trace.WriteLine("Processing message: " + sendEmail.MessageId);
System.Threading.Thread.Sleep(r.Next(10000));
// Mark it as completed
sendEmail.BeginComplete(ProcessEndComplete, sendEmail);
}
}
void ProcessEndComplete(IAsyncResult result)
{
var bm = result.AsyncState as BrokeredMessage;
bm.EndComplete(result);
Trace.WriteLine("Completed message: " + bm.MessageId);
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
// Create the queue if it does not exist already
string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.QueueExists(QueueName))
{
namespaceManager.CreateQueue(QueueName);
}
// Initialize the connection to Service Bus Queue
Client = QueueClient.CreateFromConnectionString(connectionString, QueueName);
IsStopped = false;
return base.OnStart();
}
public override void OnStop()
{
// Waiting for all requestes to finish (or timeout) before closing
while (dequeueRequests > 0)
{
System.Diagnostics.Trace.WriteLine(dequeueRequests + " request(s), waiting before stopping");
Thread.Sleep(2000);
}
// Close the connection to Service Bus Queue
IsStopped = true;
Client.Close();
base.OnStop();
}
}
}
Hope it helps.