We Keep Coding

Mass Transit DiscardSkippedMessages on Azure Service Bus topic - how to completly ignore unknown message types

I am struggling to understand the DiscardSkippedMessages setting on my ASB topic subscription endpoint.
Summary:
From what I understand, if the message type is unknown to the subscriber or has no consumers registered, the message is skipped. According to MT docs, this message should go to "xxx_skipped" queue by default. I should be able to adjust this behavour to complelty discard it, or raise an exception.
I may misunderstand what "discard" means in this context. I'd expect that the message is acknowledged and forgotten, however it goes to the dead letter queuee instead.
Context:
I am working on a solution where we have already defined topics. Single topic may have multiple message types. MT supports this without any issues, however this architecture raises a problem when it comes to future deployments. When a publishing application sends a new event, all consumers that happen to be subscribed to given topic will need to have an empty handler:
subConfig.Handler<EventIAmNotInterestedIn>(_ => Task.CompletedTask);
We want to stick to the current architectire where single topic can have multiple releted event types. This is why I started to investigate how to complelty ignore any message that has no known consumers. DiscardSkippedMessages seemed like exectly what I wanted.
All Fault<> are setup to land in a different topic.
We are using version 7.3.0 of Mass Transit.
Question 1:
Difference between _skipped queue and dead letter queue in Azure Service Bus.
I'd assume that default configuration of MT would result in this queue being created, however the skipped messages are going to dead letter. Does this implies, that _skipped queue is a thing in other thansports, but for ASB it is dead letter?
public static async Task Main(string[] args)
{
await Host.CreateDefaultBuilder(args)
.UseSerilog((context, services, configuration) =>
{
configuration.MinimumLevel.Debug();
configuration.WriteTo.Console();
})
.ConfigureServices(services =>
{
services.AddMassTransit(mtConfig =>
{
mtConfig.AddConsumers(Assembly.GetExecutingAssembly());
// minimal working configuration
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
});
services.AddHostedService<BusHostedService>();
})
.Build()
.RunAsync();
}
where BusHostedService is used only to start/stop the bus. I am submitting messages manually with Azure Service Bus Explorer.
internal class BusHostedService : IHostedService
{
private readonly IBusControl _bus;
public BusHostedService(IBusControl bus)
{ _bus = bus; }
public Task StartAsync(CancellationToken cancellationToken) => _bus.StartAsync(cancellationToken);
public Task StopAsync(CancellationToken cancellationToken) => _bus.StopAsync(cancellationToken);
}
When I submit a message with a unknown type it is skipped as expected:
{
"messageId": "e934af95-33ea-4df0-a793-62f9e488e78d",
"correlationId": "75694f2c-ea53-4568-887b-1b5abbade1c2",
"conversationId": "66700000-cd86-922e-fc03-08da7096c7ea",
"sourceAddress": "sb://***.servicebus.windows.net/***",
"destinationAddress": "sb://***.servicebus.windows.net/***",
"messageType": [
"urn:message:SomeNamespace:UnknownEvent" <---- this type does not exist and have no consumer
],
"message": { },
"sentTime": "2022-07-28T12:43:32.7704944Z",
"headers": { },
"host": {
"machineName": "6a13b9ccde39",
"processName": "***",
"processId": 8,
"assembly": "***",
"assemblyVersion": "1.0.603.0",
"frameworkVersion": "6.0.7",
"massTransitVersion": "7.3.0.0",
"operatingSystemVersion": "Unix 5.4.0.1074"
}
}
In logs I can see that it was skipped:
[11:12:57 DBG] Hosting starting
[11:12:58 INF] Configured endpoint testsub, Consumer: MTProducer.SomeEventConsumer
[11:12:58 DBG] Starting bus: ***
[11:12:59 DBG] Endpoint Ready: ****
[11:13:00 DBG] Topic: some.topic ()
[11:13:00 DBG] Subscription testsub (some.topic -> null)
[11:13:00 DBG] Endpoint Ready: sb://***.servicebus.windows.net/some.topic/Subscriptions/testsub
[11:13:00 INF] Bus started: sb://***.servicebus.windows.net/
*[11:13:13 DBG] SKIP sb://***.servicebus.windows.net/some.topic/Subscriptions/testsub 54d1e792-0ea6-42a0-b93f-1fa8d1d3b3d4*
and I can see that it landed in dead letter, no _skipped queue was created
screenshot showing a topic and subscription with single item in dead letter queue
So, does this mean that for ASB the default way of dealing with skipped messages is to put them into dead letter queue?
Question 2:
Since we already established what default configuration does, I want to discard this message. I'd like to have it complelty forgotten and not sent into dead letter.
I will do it with DiscardSkippedMessages method.
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.DiscardSkippedMessages();
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
After submitting another message, I also can see in the logs, that it was skipped, but it is also delivered to dead letter queue again.
screenshot showing a topic and subscription with two items in dead letter queue
So, does this mean that this setting is not accepted by ASB transport? I do not see any change in behaviour, especially the one I expect.
Is it possible to "trully discard" a message that I am not interested in?
Possible workaround:
I was able to implement desired behaviour with a custom filter that uses ReceiveContext.IsDelivered to determine whenever the message was consumed by anyone.
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.ConfigureDeadLetter(x =>
{
x.UseFilter(new CustomDiscardDeadLetterFilter());
});
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
internal class CustomDiscardDeadLetterFilter : IFilter<ReceiveContext>
{
public void Probe(ProbeContext context) => context.CreateFilterScope("custom-discard-dead-letter");
public Task Send(ReceiveContext context, IPipe<ReceiveContext> next)
{
if (!context.IsFaulted && !context.IsDelivered)
{
return Task.CompletedTask;
}
return next.Send(context);
}
}
I have my doubts about this workaround and this needs to be tested for issues when there are also retries, other filters involved in the pipeline.
Is this workaround close to a proper solution of ignoring unknown messages, or I am missing a lot here?
Question 3:
Is redesigning our existing eventing architecture to have a single event type per topic the only option to cleanly solve this?

To answer your multiple questions:
Subscription endpoints don't have queues, so there is no _error or _skipped queue. By default, all skipped or faulted messages are moved to the DLQ.
On normal endpoints, this behavior is configurable. However, because the subscription endpoint already replaces the default empty pipes for skipped/error with filters that move to the DLQ, the pipe is never empty and the discard filter is added after the move filter. So, short answer, it's a limitation of the current codebase. I can add an issue, but it will only be fixed in v8.
It certainly isn't the only way. The only reason they go to the DLQ is because they're skipped. And the reason should show that (versus being faulted). I can't say that I've built a solution that pushes multiple types into a single topic, but plenty of people have and manage to do it just fine. So, just like everything else in software, it depends.

MassTransit not subscribing to AzureServiceBus Topic

I'm currently trying to update application that was originally .NET Core 3.1 using MassTransit 6.3.2. It is now configured to use .NET 6.0 and MassTransit 7.3.0
Our application uses MassTransit to send messages via Azure Service Bus, publishing messages to Topics, which then have other Subscribers listening to those Topic.
Cut down, it was implemented like so:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer>();
config.AddBus(BusControlFactory.ConfigureAzureServiceBus);
});
// BusControlFactory.cs
public static class BusControlFactory
{
public static IBusControl ConfigureAzureServiceBus(IRegistrationContext<IServiceProvider> context)
{
var config = context.Container.GetService<AppConfiguration>();
var azureServiceBus = Bus.Factory.CreateUsingAzureServiceBus(busFactoryConfig =>
{
busFactoryConfig.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
busFactoryConfig.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
busFactoryConfig.SubscriptionEndpoint<AppointmentBookedMessage>(
"my-subscriber-name",
configurator =>
{
configurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
configurator.Consumer<AppointmentBookedMessageConsumer>(context.Container);
});
return azureServiceBus;
}
}
}
It has now been changed and upgraded to the latest MassTransit and is implemented like:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer, AppointmentBookedMessageConsumerDefinition>();
config.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
cfg.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
cfg.ConfigureEndpoints(context);
});
// AppointmentBookedMessageConsumerDefinition.cs
public class AppointmentBookedMessageConsumerDefinition: ConsumerDefinition<AppointmentBookedMessageConsumer>
{
public AppointmentBookedMessageConsumerDefinition()
{
EndpointName = "testharness.subscriber";
}
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator, IConsumerConfigurator<AppointmentBookedMessageConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
}
}
The issue if it can be considered one, is that I can't bind to a subscription that already exists.
In the example above, you can see that the EndpointName is set as "testharness.subscriber". There was already a subscription to the Topic "appointment-booked" from prior to me upgrading. However, when the application runs, it does not error, but it receives no messages.
If I change the EndpointName to "testharness.subscriber2". Another subscriber appears in the Azure Service Bus topic (via the Azure Portal) and I start receiving messages. I can see no difference in the names (other than the change that I placed, in this case: the "2" suffix).
Am I missing something here? Is there something else I need to do to get these to bind? Is my configuration wrong? Was it wrong? While I'm sure I can get around this by managing the release more closely and removing unneeded queues once they're using new ones - it feels like the wrong approach.

With Azure Service Bus, ForwardTo on a subscription can be a bit opaque.
While the subscription may indeed visually indicate that it is forwarding to the correctly named queue, it might be that the queue was deleted and recreated at some point without deleting the subscription. This results in a subscription that will build up messages, as it is unable to forward them to a queue that no longer exists.
Why? Internally, a subscription maintains the ForwardTo as an object id, which after the queue is deleted points to an object that doesn't exist – resulting in messages building up in the subscription.
If you have messages in the subscription, you may need to go into the portal and update that subscription to point to the new queue (even though it has the same name), at which point the messages should flow through to the queue.
If there aren't any messages in the subscription (or if they aren't important), you can just delete the subscription and it will be recreated by MassTransit when you restart the bus.

Rebus - Subscribing to Events in ASP.NET Core

I have two applications using Rebus in ASP.NET MVC Core
I am able send messages between two applications using Bus.Send(...). What I can't is to publish event such as CustomerCreated after creating so that other applications can take actions.
I have configured the application as follows
public void ConfigureServices(IServiceCollection services)
{
services.AutoRegisterHandlersFromAssemblyOf<Handler1>();
services.AddRebus(configure => configure
.Logging(l => l.Use(new MSLoggerFactoryAdapter(_loggerFactory)))
.Transport(t=>t.UseRabbitMq("amqp://guest:guest#localhost:5672", "rebus_rabbit_first"))
.Sagas(x => x.StoreInSqlServer("Data Source=.;Initial Catalog=RebusDBRabbit;User ID=student;Password=student;", "Sagas", "SagaIndex"))
.Options(o =>
{
o.SetNumberOfWorkers(10);
o.SetMaxParallelism(20);
o.HandleMessagesInsideTransactionScope();
o.SimpleRetryStrategy(errorQueueAddress: "somewhere_else", maxDeliveryAttempts: 10, secondLevelRetriesEnabled: true);
})
.Routing(r => r.TypeBased()
.MapAssemblyOf<CreateStudent>("rebus_rabbit_second")));
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
}
In the Controller I Send a message to another Application as follows
CreateStudent student = new CreateStudent { StudentID="90008", StudentName="Amour Rashid Hamad",DateOfBirth=DateTime.Parse("1974-03-18") };
_bus.Send(student).Wait();
This is OK.
Now My Problem is to publish an event to broadcast the event to other interested parties eg
_bus.Publish(new StudentCreated { StudentID="90008",Remarks="Hurray We have been Successfully"});
How Do I Subscribe to the event as per my configuration. I have seen some samples but I could not understand them. Adding to my implementation would be preferred.

In the Service Configuration I did as follows:
app.ApplicationServices.UseRebus(async bus => {
await bus.Subscribe<StudentCreated>();
});
and then created a handler
public class StudentCreatedEventHandler : IHandleMessages<StudentCreated>, IHandleMessages<IFailed<StudentCreated>>
{
readonly IBus _bus;
public StudentCreatedEventHandler(IBus bus)
{
_bus = bus;
}
public async Task Handle(StudentCreated student)
{
// do stuff that can fail here...
var remarks = $"Remarks on RebusWithRabbit1 : {student.Remarks}";
}
public async Task Handle(IFailed<StudentCreated> failedMessage)
{
await _bus.Advanced.TransportMessage.Defer(TimeSpan.FromSeconds(30));
}
}
This could handle the events published.
I just want to get assured if that is the proper way of doing it.
I have however noticed one thing. If I have more than one endpoints Subscribing to the event only one is notified. I expected that multiple endpoints could need to be notified and every one may execute a different process from the same event.
Is there any way to change this behavior. I remember in MassTransit this is the default behavious.
Thanks

It looks like you're using await bus.Send(...) properly.
As you've probably figured out, Send looks up a destination queue and sends to that (and only that), and the lookup is done from the endpoint mappings (where you're currently mapping all message types to the queue named rebus_rabbit_second).
When you want to await bus.Publish(...), you need someone to await bus.Subscribe<T>() accordingly. Underneath the covers, Rebus will use the .NET type as the topic, so if you
await bus.Subscribe<SomeEvent>();
in one application, and
await bus.Publish(new SomeEvent());
in another, your subscriber will receive the event.
TL;DR: You need to
await bus.Subscribe<StudentCreated>();
in the application where you want to receive published events of type StudentCreated.
Btw. you should EITHER use C#'s support for calling asynchronous methods
await bus.Send(yourMessage);
or invoke Rebus' synchronous API to do your work:
var syncBus = bus.Advances.SyncBus;
syncBus.Send(yourMessage); //< returns void
Rebus' synchronous methods do not deadlock the thread, e.g. if you're calling them from ASP.NET or WCF.

MassTransit saga with request/response timeouts when deployed to test server

I have a fully working MassTransit saga, which runs some commands and then executes a request/response call to query a database and then ultimately return a response to the calling controller.
Locally this all works now 99% of the time (thanks to a lot of support I've received on here). However, when deployed to my Azure VM, which has a local copy of RabbitMQ and the 2 ASP.NET Core services running on it, the first call to the saga goes through straight away but all subsequent calls timeout.
I feel like it might be related to the fact that I'm using an InMemorySagaRepository (which in theory should be fine for my use case).
The saga is configured initially like so:
InstanceState(s => s.CurrentState);
Event(() => RequestLinkEvent, x => x.CorrelateById(context => context.Message.LinkId));
Event(() => LinkCreatedEvent, x => x.CorrelateById(context => context.Message.LinkId));
Event(() => CreateLinkGroupFailedEvent, x => x.CorrelateById(context => context.Message.LinkId));
Event(() => CreateLinkFailedEvent, x => x.CorrelateById(context => context.Message.LinkId));
Event(() => RequestLinkFailedEvent, x => x.CorrelateById(context => context.Message.LinkId));
Request(() => LinkRequest, x => x.UrlRequestId, cfg =>
{
cfg.ServiceAddress = new Uri($"{hostAddress}/{nameof(SelectUrlByPublicId)}");
cfg.SchedulingServiceAddress = new Uri($"{hostAddress}/{nameof(SelectUrlByPublicId)}");
cfg.Timeout = TimeSpan.FromSeconds(30);
});
It's worth noting that my LinkId is ALWAYS a unique Guid as it is created in the controller before the message is sent.
ALSO when I restart the apppool it works again for the first call and then starts timing out again.
I feel like something might be locking somewhere but I can't reproduce it locally!

So I wanted to post my solution to my own problem here in the hopes that it will aide others in the future.
I made 3 fundamental changes which either in isolation or combination solved this issue and everything now flys and works 100% of the time whether I use an InMemorySagaRepository, Redis or MongoDB.
Issue 1
As detailed in another question I posted here:
MassTransit saga with Redis persistence gives Method Accpet does not have an implementation exception
In my SagaStateMachineInstance class I had mistakenly declared the CurrentState property as a 'State' type when it should have been a string as such:
public string CurrentState { get; set;}
This was a fundamental issue and it came to light as soon as I started trying to add persistence so it may have been causing troubles when using the InMemorySagaRepository too.
Issue 2
In hindsight I suspect this was probably my main issue and I'm not completely convinced I've solved it in the best way but I'm happy with how things are.
I made sure my final event is managed in all states. I think what was happening was my request/response was finishing before the CurrentState of the saga had been updated. I realised this was happening by experimenting with using MongoDB as my persistence and seeing that I had sagas not completing stuck in the penultimate state.
Issue 3
This should be unnecessary but I wanted to add it as something to consider/try for those having issues.
I removed the request/response step from my saga and replaced it with a publish/subscribe. To do this I published an event to my consumer which when complete publishes an event with the CorrelationId (as suggested by #alexey-zimarev in my other issue). So in my consumer that does the query (i.e. reuqest) I do the following after it completes:
context.Publish(new LinkCreatedEvent { ... , CorrelationId = context.Message.CorrelationId })
Because the CorrelationId is in there my saga picks it up and handles the event as such:
When(LinkCreatedEvent )
.ThenAsync(HandleLinkCreatedEventAsync)
.TransitionTo(LinkCreated)
I'm really happy with how it all works now and feel confident about putting the solution live.

How to log if nservicebus saga is already started

I have a saga:
public class MySaga : Saga<MySagaEntity>,
IAmStartedByMessages<Message1>,
IAmStartedByMessages<Message2> {
}
In general I need to see easily from logs which of the messages starts which saga.
What I need is to log something like :
Recieved message Message1 with ... which starts a new saga
recieved message Message2 with ... for exsisting saga wiht Id=...
As alternatives i have following ways:
1. check if log file if that saga was not started
2. check if correlationid of saga is empty (so as it will be filled within handlers which start the saga)
if (Data.CorrelationId == default_value)
_log.DebugFormat("message starts saga CorrelationId={0}", message.CorrelationId)
Does anyone knew better ways for this?

There isn't currently a way in NServiceBus to get notified if a saga has been created or if a existing instance was loaded. (I've opened up a github issue for further discussion)
That said if the fact that the saga was created by a given message has a business meaning you're probably better off setting a boolean flag on your saga data to record this explicitly.
if(Data.SagaWasStartedByAOnlineCustomer)
Bus.Send(new VerifySomethingForOnlineCustomersCommand);