We Keep Coding

Mass Transit RabbitMQ Consumer fails with long task

Error Message: MassTransit.ConnectionException: The connection is stopping and cannot be used: rabbit-host
We have a long-running consumer using MassTransit with RabbitMQ.
We are failing in the consumer when trying to publish our result onto a different queue after running for 20+ minutes.
We are assuming that the connection is timing out before we complete our work.
We see there is an option to use a JobConsumer for long running tasks, but were wondering if there is a way to extend our timeout on the regular Consumer when working with RabbitMQ?
We saw the MaxAutoRenewDuration option when working with Azure on this question: Masstransit - long running process and imediate response and were looking for something similar for RabbitMQ.
Is there a specific default timeout time that the connection to the rabbit host lasts?
Thank you for any help, and I can provide more details if that'd be helpful.
// Consumer Class
using System;
using System.Threading.Tasks;
using MassTransit;
namespace ExampleNameSpace
{
public class ExampleConsumer : IConsumer<ExampleMessage>
{
private readonly BusinessLogicProcess _businessLogicProcess;
public ExampleConsumer(BusinessLogicProcess businessLogicProcess)
{
_businessLogicProcess = businessLogicProcess;
}
public async Task Consume(ConsumeContext<ExampleMessage> context)
{
try
{
// Do our business logic (takes 20+ minutes)
var businessLogicResult = await _businessLogicProcess.DoBusinessWorkAsync();
var resultMessage = new ResultMessage { ResultValue = businessLogicResult };
// Publish the result of our work
// Get the following error when we publish after a long running piece of work
// MassTransit.ConnectionException: The connection is stopping and cannot be used: rabbitmqs://our-rabbit-host/vhost-name
await context.Publish<ResultMessage>(resultMessage);
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
}
public class BusinessLogicProcess {
public async Task<int> DoBusinessWorkAsync()
{
// Business Logic Here
// Takes 20+ minutes
return 0;
}
}
public class ExampleMessage { }
public class ResultMessage {
public int ResultValue { get; set; }
}
}
// Service Class - Connect to RabbitMQ
using MassTransit;
using Microsoft.Extensions.Hosting;
using System;
using System.Threading;
using System.Threading.Tasks;
namespace ExampleNameSpace
{
public class ExampleService : BackgroundService
{
private readonly BusinessLogicProcess _businessLogicProcess;
private IBusControl _bus;
public ExampleService(BusinessLogicProcess process)
{
_businessLogicProcess = process;
}
public override async Task StartAsync(CancellationToken cancellationToken)
{
_bus = Bus.Factory.CreateUsingRabbitMq(
config =>
{
config.Host(new Uri("rabbitmqs://our-rabbit-host/vhost-name"), hostConfig =>
{
hostConfig.Username("guest");
hostConfig.Password("guest");
});
config.ReceiveEndpoint("exampleendpoint",
endpointConfigurator =>
{
endpointConfigurator.Consumer(() => new ExampleConsumer(_businessLogicProcess),
config => config.UseConcurrentMessageLimit(1));
});
});
await _bus.StartAsync(cancellationToken);
await base.StopAsync(cancellationToken);
}
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
return Task.CompletedTask;
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
await _bus.StopAsync(cancellationToken);
await base.StopAsync(cancellationToken);
}
}
}

Azure Service Bus Topics and Subscriptions with Worker Role

So I've recently gotten the need to use Service Bus Topic and Subscriptions and I've followed many articles and tutorials. I've been able to successfully implement Microsoft's Get started with Service Bus topics and also successfully used Visual Studio 2017's Worker Role template to access a database.
However, I'm confused as to how to properly "combine" the two. While the Get started with Service Bus topics article shows how to create 2 apps, one to send and one to receive and then quits, the Worker Role template seems to loops endlessly with await Task.Delay(10000);.
I'm not sure how to "mesh" the two properly. Essentially, I want my Worker Role to stay alive and listen for entries into it's subscription forever (or until it quits obviously).
Any guidance would be great!
P.S.: I've asked a related question concerning proper technology I should use for my case scenario at StackExchange - Software Engineering if you are interested.
Update #1 (2018/08/09)
Based on Arunprabhu's answer, here is some code of how I'm sending a Message based on articles I've read and receiving using Visual Studio 2017's Worker Role with Service Bus Queue template.
Sending (based on Get started with Service Bus topics)
using System;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
namespace TopicsSender {
internal static class Program {
private const string ServiceBusConnectionString = "<your_connection_string>";
private const string TopicName = "test-topic";
private static ITopicClient _topicClient;
private static void Main(string[] args) {
MainAsync().GetAwaiter().GetResult();
}
private static async Task MainAsync() {
const int numberOfMessages = 10;
_topicClient = new TopicClient(ServiceBusConnectionString, TopicName);
Console.WriteLine("======================================================");
Console.WriteLine("Press ENTER key to exit after sending all the messages.");
Console.WriteLine("======================================================");
// Send messages.
await SendMessagesAsync(numberOfMessages);
Console.ReadKey();
await _topicClient.CloseAsync();
}
private static async Task SendMessagesAsync(int numberOfMessagesToSend) {
try {
for (var i = 0; i < numberOfMessagesToSend; i++) {
// Create a new message to send to the topic
var messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
// Write the body of the message to the console
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the topic
await _topicClient.SendAsync(message);
}
} catch (Exception exception) {
Console.WriteLine($"{DateTime.Now} :: Exception: {exception.Message}");
}
}
}
}
Receiving (based on Worker Role with Service Bus Queue template)
using System;
using System.Diagnostics;
using System.Net;
using System.Threading;
using Microsoft.ServiceBus.Messaging;
using Microsoft.WindowsAzure.ServiceRuntime;
namespace WorkerRoleWithSBQueue1 {
public class WorkerRole : RoleEntryPoint {
// The name of your queue
private const string ServiceBusConnectionString = "<your_connection_string>";
private const string TopicName = "test-topic";
private const string SubscriptionName = "test-sub1";
// QueueClient is thread-safe. Recommended that you cache
// rather than recreating it on every request
private SubscriptionClient _client;
private readonly ManualResetEvent _completedEvent = new ManualResetEvent(false);
public override void Run() {
Trace.WriteLine("Starting processing of messages");
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
_client.OnMessage((receivedMessage) => {
try {
// Process the message
Trace.WriteLine("Processing Service Bus message: " + receivedMessage.SequenceNumber.ToString());
var message = receivedMessage.GetBody<byte[]>();
Trace.WriteLine($"Received message: SequenceNumber:{receivedMessage.SequenceNumber} Body:{message.ToString()}");
} catch (Exception e) {
// Handle any message processing specific exceptions here
Trace.Write(e.ToString());
}
});
_completedEvent.WaitOne();
}
public override bool OnStart() {
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
// Initialize the connection to Service Bus Queue
_client = SubscriptionClient.CreateFromConnectionString(ServiceBusConnectionString, TopicName, SubscriptionName);
return base.OnStart();
}
public override void OnStop() {
// Close the connection to Service Bus Queue
_client.Close();
_completedEvent.Set();
base.OnStop();
}
}
}
Update #2 (2018/08/10)
After a few suggestions from Arunprabhu and knowing I was using different libraries, below is my current solution with pieces taken from several sources. Is there anything I'm overlooking, adding that shouldering be there, etc? Currently getting an error that may be for another question or already answered so don't want to post it yet before further research.
using System;
using System.Diagnostics;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
using Microsoft.WindowsAzure.ServiceRuntime;
namespace WorkerRoleWithSBQueue1 {
public class WorkerRole : RoleEntryPoint {
private readonly CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();
private readonly ManualResetEvent _runCompleteEvent = new ManualResetEvent(false);
// The name of your queue
private const string ServiceBusConnectionString = "<your_connection_string>";
private const string TopicName = "test-topic";
private const string SubscriptionName = "test-sub1";
// _client is thread-safe. Recommended that you cache
// rather than recreating it on every request
private SubscriptionClient _client;
public override void Run() {
Trace.WriteLine("Starting processing of messages");
try {
this.RunAsync(this._cancellationTokenSource.Token).Wait();
} catch (Exception e) {
Trace.WriteLine("Exception");
Trace.WriteLine(e.ToString());
} finally {
Trace.WriteLine("Finally...");
this._runCompleteEvent.Set();
}
}
public override bool OnStart() {
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
var result = base.OnStart();
Trace.WriteLine("WorkerRole has been started");
return result;
}
public override void OnStop() {
// Close the connection to Service Bus Queue
this._cancellationTokenSource.Cancel();
this._runCompleteEvent.WaitOne();
base.OnStop();
}
private async Task RunAsync(CancellationToken cancellationToken) {
// Configure the client
RegisterOnMessageHandlerAndReceiveMessages(ServiceBusConnectionString, TopicName, SubscriptionName);
_runCompleteEvent.WaitOne();
Trace.WriteLine("Closing");
await _client.CloseAsync();
}
private void RegisterOnMessageHandlerAndReceiveMessages(string connectionString, string topicName, string subscriptionName) {
_client = new SubscriptionClient(connectionString, topicName, subscriptionName);
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler) {
// Maximum number of concurrent calls to the callback ProcessMessagesAsync(), set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 1,
// Indicates whether MessagePump should automatically complete the messages after returning from User Callback.
// False below indicates the Complete will be handled by the User Callback as in `ProcessMessagesAsync` below.
AutoComplete = false,
};
_client.RegisterMessageHandler(ProcessMessageAsync, messageHandlerOptions);
}
private async Task ProcessMessageAsync(Message message, CancellationToken token) {
try {
// Process the message
Trace.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
await _client.CompleteAsync(message.SystemProperties.LockToken);
} catch (Exception e) {
// Handle any message processing specific exceptions here
Trace.Write(e.ToString());
await _client.AbandonAsync(message.SystemProperties.LockToken);
}
}
private static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs) {
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}

Considering the complexity of the updated question Update #1 (2018/08/09), I am providing a separate answer.
The sender and receiver are using different libraries.
Sender - Microsoft.Azure.ServiceBus
Receiver - WindowsAzure.ServiceBus
Microsoft.Azure.ServiceBus has the message object as Message, where WindowsAzure.ServiceBus has BrokeredMessage.
There is a method RegisterMessageHandler available in Microsoft.Azure.ServiceBus, this is the alternative for client.OnMessage() in WindowsAzure.ServiceBus. By using this, the listener receives the message as Message object. This library supports asynchronous programming as you expect.
Refer here for samples from both the libraries.

If you are using Visual Studio, there is a default template available for creating Azure Cloud Service and Worker Role with Service Bus Queue. There you need to change the QueueClient with SubscriptionClient in WorkerRole.cs.
Then, the worker role will stay active, listening for the messages from Topic Subscription.
You can find the samples here. You should create Worker role with Service Bus Queue inside the Cloud Service

ASB MessageReceiver ReceiveAsync crashes

Environment
Windows 10 Professional
.NET Core Console Application
Code
I have an abstracted message receiver that looks like this. In this code the entity is the name of the Subscription (e.g. user).
public class AzureMessageReceiver : ITdlMessageReceiver
{
private readonly ServiceBusConnection serviceBusConnection;
private readonly ILogger<AzureMessageReceiver> logger;
public AzureMessageReceiver(ServiceBusConnection serviceBusConnection, ILogger<AzureMessageReceiver> logger)
{
this.serviceBusConnection = serviceBusConnection;
this.logger = logger;
}
public async Task<TdlMessage<T>> ReceiveAsync<T>(string topic, string entity) where T : class
{
try
{
var subscriptionPath = EntityNameHelper.FormatSubscriptionPath(topic, entity);
var messageReceiver = new MessageReceiver(serviceBusConnection, subscriptionPath, ReceiveMode.ReceiveAndDelete);
var message = await messageReceiver.ReceiveAsync();
if (message == null)
{
return null;
}
var messageString = Encoding.UTF8.GetString(message.Body);
return JsonConvert.DeserializeObject<TdlMessage<T>>(messageString);
}
catch (Exception ex)
{
logger.LogError(ex, "Error receiving Azure message.");
return null;
}
}
}
The injected ServiceBusConnection is constructed like this. NOTE: this same connection initialization works to write messages to the same Topic and Subscription.
services.AddSingleton(serviceProvider =>
new ServiceBusConnection(configuration[$"{DurableCommunicationKey}:AzureConnectionString"]));
UPDATE: here is the code that wraps the call to the receiver class and is the controller for receiving messages:
static async void Receive(ITdlMessageReceiver receiver, ILogger logger)
{
while (true)
{
var message = await receiver.ReceiveAsync<TdlMessage<object>>(topic, entity);
if (message != null)
{
logger.LogDebug($"Message received. Topic: {topic}. Action: {Enum.GetName(typeof(TopicActions), message.Action)}. Message: {JsonConvert.SerializeObject(message)}.");
}
Thread.Sleep(sleepTime);
}
}
Problem
Every time I execute this line var message = await messageReceiver.ReceiveAsync(); it just crashes the Console app. No Exception and nothing in Event Viewer.
What I've Tried
Using the Secondary Connection String from the ASB
Providing a timeout like messageReceiver.ReceiveAsync(TimeSpan.FromMinutes(1));
Changing the injected topic from just the name of the topic to the entire URL of the topic (e.g. https://{...}.servicebus.windows.net/{topicName})
Changing the ReceiveMode to PeekLock
Tacking on ConfigureAwait(false) to the ReceiveAsync call.
Changing the timeout to TimeSpan.Zero. NOTE: this does not crash the app but actually throws an Exception that gets logged.

async void should be converted to an async Task as well as you should be awaiting Task.Delay instead of invoking Thread.Sleep. If going async you need to go async all the way
static async Task Receive(ITdlMessageReceiver receiver, ILogger logger) {
while (true) {
var message = await receiver.ReceiveAsync<TdlMessage<object>>(topic, entity);
if (message != null) {
logger.LogDebug($"Message received. Topic: {topic}. Action: {Enum.GetName(typeof(TopicActions), message.Action)}. Message: {JsonConvert.SerializeObject(message)}.");
}
await Task.Delay(sleepTime);
}
}
Try making the code async all the way through, yes, but as a console application (single thread) you will be allowed to call Wait() on the Receive method in Main as it is not mixing calls that would cause problem with the async flow.
public static void Main(string[] args) {
//...
//...
//...
Receive(receiver, logger).Wait();
}
Reference Async/Await - Best Practices in Asynchronous Programming

ETW filter by event ID using TraceEvent

I'm emitting the ETW events using the Microsoft.Diagnostics.Tracing.EventSource nuget version 1.1.28 and tracing the events on the fly using the Microsoft.Diagnostics.Tracing.TraceEvent nuget version 1.0.41. What I'm trying to do is filter the events that will be traced by the event id and I can't seem to get it to work, so I've created a simple test project to test this out on a machine with Windows 10 Enterprise using .NET framework version 4.6
My event source code:
[EventSource(Name = ProviderName)]
public sealed class EtwDemoEventSource : EventSource
{
private static Lazy<EtwDemoEventSource> instance = new Lazy<EtwDemoEventSource>(() => new EtwDemoEventSource());
public const string ProviderName = "EtwDemoEventSource";
private EtwDemoEventSource() { }
public static EtwDemoEventSource Log
{
get
{
return instance.Value;
}
}
[Event(1, Message = "Foo event raised.")]
public void FooEvent()
{
WriteEvent(1);
}
[Event(2, Message = "Bar event raised.")]
public void BarEvent()
{
WriteEvent(2);
}
}
I have one console app which emits these events using the event source:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Start emitting events.");
EtwDemoEventSource.Log.FooEvent();
EtwDemoEventSource.Log.BarEvent();
Console.WriteLine("Finish emitting events.");
Console.ReadLine();
}
}
And another app which traces the events:
class Program
{
static void Main(string[] args)
{
using (var session= new TraceEventSession("EtwTraceEventSession"))
{
Console.WriteLine("Start collecting events.");
session.Source.Dynamic.All += te => Console.WriteLine(te.FormattedMessage);
session.EnableProvider(EtwDemoEventSource.ProviderName, options: new TraceEventProviderOptions { EventIDsToEnable = new int[] { 1 } });
Console.CancelKeyPress += (sender, cargs) => session.Stop();
session.Source.Process();
Console.WriteLine("Finish collecting events.");
}
}
}
If you look at the EnableProvider method in the trace app, I've found that the TraceEventProviderOptions has an EventIDsToEnable property which seems to suit my use case, but running the trace app with that code (first I run the trace app, then the app which emits the events) produces no events traced. If I remove the TraceEventProviderOptions then all the events are traced properly, also if I use the EventIDsToDisable property it works as expected.
Have I made an error in assuming how EventIDsToEnable property should affect event filtering (I can't seem to find much documentation on it) or should I be using it in a different manner?

Asynchronous insert in Azure Table

How to asynchronously save an entity to Windows Azure Table Service?
The code below works synchronously but raises an exception when trying to save asynchronously.
This statement:
context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
(asyncResult => context.EndSaveChanges(asyncResult)), null);
Results in System.ArgumentException: "The current object did not originate the async result. Parameter name: asyncResult".
Additionally, what's the correct pattern for creating the service context when saving asynchronously? Should I create a separate context for each write operation? Is it too expensive (e.g. requiring a call over the network)?
TableStorageWriter.cs:
using System;
using System.Data.Services.Client;
using System.Diagnostics;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;
namespace WorkerRole1
{
public class TableStorageWriter
{
private const string _tableName = "StorageTest";
private readonly CloudStorageAccount _storageAccount;
private CloudTableClient _tableClient;
public TableStorageWriter()
{
_storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
_tableClient = _storageAccount.CreateCloudTableClient();
_tableClient.CreateTableIfNotExist(_tableName);
}
public void Write(string message)
{
try
{
DateTime now = DateTime.UtcNow;
var entity = new StorageTestEntity
{
Message = message,
PartitionKey = string.Format("{0:yyyy-MM-dd}", now),
RowKey = string.Format("{0:HH:mm:ss.fff}-{1}", now, Guid.NewGuid())
};
// Should I get this context before each write? It is efficient?
TableServiceContext context = _tableClient.GetDataServiceContext();
context.AddObject(_tableName, entity);
// This statement works but it's synchronous
context.SaveChangesWithRetries();
// This attempt at saving asynchronously results in System.ArgumentException:
// The current object did not originate the async result. Parameter name: asyncResult
// context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
// (asyncResult => context.EndSaveChanges(asyncResult)), null);
}
catch (StorageClientException e)
{
Debug.WriteLine("Error: {0}", e.Message);
Debug.WriteLine("Extended error info: {0} : {1}",
e.ExtendedErrorInformation.ErrorCode,
e.ExtendedErrorInformation.ErrorMessage);
}
}
}
internal class StorageTestEntity : TableServiceEntity
{
public string Message { get; set; }
}
}
Called from WorkerRole.cs:
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure.ServiceRuntime;
using log4net;
namespace WorkerRole1
{
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
var storageWriter = new TableStorageWriter();
while (true)
{
Thread.Sleep(10000);
storageWriter.Write("Working...");
}
}
public override bool OnStart()
{
ServicePointManager.DefaultConnectionLimit = 12;
return base.OnStart();
}
}
}
Examples using Windows Azure SDK for .NET 1.8.

You should call EndSaveChangesWithRetries instead of EndSaveChanges, as otherwise the IAsyncResult object returned by BeginSaveChangesWithRetries cannot be used by EndSaveChanges. So, could you please try changing your End method call as below?
context.BeginSaveChangesWithRetries(SaveChangesOptions.Batch,
(asyncResult => context.EndSaveChangesWithRetries(asyncResult)),
null);
And for your other question, I would recommend creating a new TableServiceContext for each call, as DataServiceContext is not stateless (MSDN) and the way you implemented TableStorageWriter.Write with the asynchronous call might allow concurrent operations. Actually, in Storage Client Library 2.0, we explicitly prevented concurrent operations that uses a single TableServiceContext object. Moreover, creating a TableServiceContext does not result in a request to Azure Storage.