We Keep Coding

Mass Transit DiscardSkippedMessages on Azure Service Bus topic - how to completly ignore unknown message types

I am struggling to understand the DiscardSkippedMessages setting on my ASB topic subscription endpoint.
Summary:
From what I understand, if the message type is unknown to the subscriber or has no consumers registered, the message is skipped. According to MT docs, this message should go to "xxx_skipped" queue by default. I should be able to adjust this behavour to complelty discard it, or raise an exception.
I may misunderstand what "discard" means in this context. I'd expect that the message is acknowledged and forgotten, however it goes to the dead letter queuee instead.
Context:
I am working on a solution where we have already defined topics. Single topic may have multiple message types. MT supports this without any issues, however this architecture raises a problem when it comes to future deployments. When a publishing application sends a new event, all consumers that happen to be subscribed to given topic will need to have an empty handler:
subConfig.Handler<EventIAmNotInterestedIn>(_ => Task.CompletedTask);
We want to stick to the current architectire where single topic can have multiple releted event types. This is why I started to investigate how to complelty ignore any message that has no known consumers. DiscardSkippedMessages seemed like exectly what I wanted.
All Fault<> are setup to land in a different topic.
We are using version 7.3.0 of Mass Transit.
Question 1:
Difference between _skipped queue and dead letter queue in Azure Service Bus.
I'd assume that default configuration of MT would result in this queue being created, however the skipped messages are going to dead letter. Does this implies, that _skipped queue is a thing in other thansports, but for ASB it is dead letter?
public static async Task Main(string[] args)
{
await Host.CreateDefaultBuilder(args)
.UseSerilog((context, services, configuration) =>
{
configuration.MinimumLevel.Debug();
configuration.WriteTo.Console();
})
.ConfigureServices(services =>
{
services.AddMassTransit(mtConfig =>
{
mtConfig.AddConsumers(Assembly.GetExecutingAssembly());
// minimal working configuration
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
});
services.AddHostedService<BusHostedService>();
})
.Build()
.RunAsync();
}
where BusHostedService is used only to start/stop the bus. I am submitting messages manually with Azure Service Bus Explorer.
internal class BusHostedService : IHostedService
{
private readonly IBusControl _bus;
public BusHostedService(IBusControl bus)
{ _bus = bus; }
public Task StartAsync(CancellationToken cancellationToken) => _bus.StartAsync(cancellationToken);
public Task StopAsync(CancellationToken cancellationToken) => _bus.StopAsync(cancellationToken);
}
When I submit a message with a unknown type it is skipped as expected:
{
"messageId": "e934af95-33ea-4df0-a793-62f9e488e78d",
"correlationId": "75694f2c-ea53-4568-887b-1b5abbade1c2",
"conversationId": "66700000-cd86-922e-fc03-08da7096c7ea",
"sourceAddress": "sb://***.servicebus.windows.net/***",
"destinationAddress": "sb://***.servicebus.windows.net/***",
"messageType": [
"urn:message:SomeNamespace:UnknownEvent" <---- this type does not exist and have no consumer
],
"message": { },
"sentTime": "2022-07-28T12:43:32.7704944Z",
"headers": { },
"host": {
"machineName": "6a13b9ccde39",
"processName": "***",
"processId": 8,
"assembly": "***",
"assemblyVersion": "1.0.603.0",
"frameworkVersion": "6.0.7",
"massTransitVersion": "7.3.0.0",
"operatingSystemVersion": "Unix 5.4.0.1074"
}
}
In logs I can see that it was skipped:
[11:12:57 DBG] Hosting starting
[11:12:58 INF] Configured endpoint testsub, Consumer: MTProducer.SomeEventConsumer
[11:12:58 DBG] Starting bus: ***
[11:12:59 DBG] Endpoint Ready: ****
[11:13:00 DBG] Topic: some.topic ()
[11:13:00 DBG] Subscription testsub (some.topic -> null)
[11:13:00 DBG] Endpoint Ready: sb://***.servicebus.windows.net/some.topic/Subscriptions/testsub
[11:13:00 INF] Bus started: sb://***.servicebus.windows.net/
*[11:13:13 DBG] SKIP sb://***.servicebus.windows.net/some.topic/Subscriptions/testsub 54d1e792-0ea6-42a0-b93f-1fa8d1d3b3d4*
and I can see that it landed in dead letter, no _skipped queue was created
screenshot showing a topic and subscription with single item in dead letter queue
So, does this mean that for ASB the default way of dealing with skipped messages is to put them into dead letter queue?
Question 2:
Since we already established what default configuration does, I want to discard this message. I'd like to have it complelty forgotten and not sent into dead letter.
I will do it with DiscardSkippedMessages method.
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.DiscardSkippedMessages();
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
After submitting another message, I also can see in the logs, that it was skipped, but it is also delivered to dead letter queue again.
screenshot showing a topic and subscription with two items in dead letter queue
So, does this mean that this setting is not accepted by ASB transport? I do not see any change in behaviour, especially the one I expect.
Is it possible to "trully discard" a message that I am not interested in?
Possible workaround:
I was able to implement desired behaviour with a custom filter that uses ReceiveContext.IsDelivered to determine whenever the message was consumed by anyone.
mtConfig.UsingAzureServiceBus((context, busConfig) =>
{
busConfig.Host(ConnectionString);
busConfig.SubscriptionEndpoint(
"testsub",
"some.topic",
subConfig =>
{
subConfig.ConfigureDeadLetter(x =>
{
x.UseFilter(new CustomDiscardDeadLetterFilter());
});
subConfig.ConfigureConsumer<SomeEventConsumer>(context);
});
busConfig.ConfigureEndpoints(context);
});
internal class CustomDiscardDeadLetterFilter : IFilter<ReceiveContext>
{
public void Probe(ProbeContext context) => context.CreateFilterScope("custom-discard-dead-letter");
public Task Send(ReceiveContext context, IPipe<ReceiveContext> next)
{
if (!context.IsFaulted && !context.IsDelivered)
{
return Task.CompletedTask;
}
return next.Send(context);
}
}
I have my doubts about this workaround and this needs to be tested for issues when there are also retries, other filters involved in the pipeline.
Is this workaround close to a proper solution of ignoring unknown messages, or I am missing a lot here?
Question 3:
Is redesigning our existing eventing architecture to have a single event type per topic the only option to cleanly solve this?

To answer your multiple questions:
Subscription endpoints don't have queues, so there is no _error or _skipped queue. By default, all skipped or faulted messages are moved to the DLQ.
On normal endpoints, this behavior is configurable. However, because the subscription endpoint already replaces the default empty pipes for skipped/error with filters that move to the DLQ, the pipe is never empty and the discard filter is added after the move filter. So, short answer, it's a limitation of the current codebase. I can add an issue, but it will only be fixed in v8.
It certainly isn't the only way. The only reason they go to the DLQ is because they're skipped. And the reason should show that (versus being faulted). I can't say that I've built a solution that pushes multiple types into a single topic, but plenty of people have and manage to do it just fine. So, just like everything else in software, it depends.

Rebus - Subscribing to Events in ASP.NET Core

I have two applications using Rebus in ASP.NET MVC Core
I am able send messages between two applications using Bus.Send(...). What I can't is to publish event such as CustomerCreated after creating so that other applications can take actions.
I have configured the application as follows
public void ConfigureServices(IServiceCollection services)
{
services.AutoRegisterHandlersFromAssemblyOf<Handler1>();
services.AddRebus(configure => configure
.Logging(l => l.Use(new MSLoggerFactoryAdapter(_loggerFactory)))
.Transport(t=>t.UseRabbitMq("amqp://guest:guest#localhost:5672", "rebus_rabbit_first"))
.Sagas(x => x.StoreInSqlServer("Data Source=.;Initial Catalog=RebusDBRabbit;User ID=student;Password=student;", "Sagas", "SagaIndex"))
.Options(o =>
{
o.SetNumberOfWorkers(10);
o.SetMaxParallelism(20);
o.HandleMessagesInsideTransactionScope();
o.SimpleRetryStrategy(errorQueueAddress: "somewhere_else", maxDeliveryAttempts: 10, secondLevelRetriesEnabled: true);
})
.Routing(r => r.TypeBased()
.MapAssemblyOf<CreateStudent>("rebus_rabbit_second")));
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
}
In the Controller I Send a message to another Application as follows
CreateStudent student = new CreateStudent { StudentID="90008", StudentName="Amour Rashid Hamad",DateOfBirth=DateTime.Parse("1974-03-18") };
_bus.Send(student).Wait();
This is OK.
Now My Problem is to publish an event to broadcast the event to other interested parties eg
_bus.Publish(new StudentCreated { StudentID="90008",Remarks="Hurray We have been Successfully"});
How Do I Subscribe to the event as per my configuration. I have seen some samples but I could not understand them. Adding to my implementation would be preferred.

In the Service Configuration I did as follows:
app.ApplicationServices.UseRebus(async bus => {
await bus.Subscribe<StudentCreated>();
});
and then created a handler
public class StudentCreatedEventHandler : IHandleMessages<StudentCreated>, IHandleMessages<IFailed<StudentCreated>>
{
readonly IBus _bus;
public StudentCreatedEventHandler(IBus bus)
{
_bus = bus;
}
public async Task Handle(StudentCreated student)
{
// do stuff that can fail here...
var remarks = $"Remarks on RebusWithRabbit1 : {student.Remarks}";
}
public async Task Handle(IFailed<StudentCreated> failedMessage)
{
await _bus.Advanced.TransportMessage.Defer(TimeSpan.FromSeconds(30));
}
}
This could handle the events published.
I just want to get assured if that is the proper way of doing it.
I have however noticed one thing. If I have more than one endpoints Subscribing to the event only one is notified. I expected that multiple endpoints could need to be notified and every one may execute a different process from the same event.
Is there any way to change this behavior. I remember in MassTransit this is the default behavious.
Thanks

It looks like you're using await bus.Send(...) properly.
As you've probably figured out, Send looks up a destination queue and sends to that (and only that), and the lookup is done from the endpoint mappings (where you're currently mapping all message types to the queue named rebus_rabbit_second).
When you want to await bus.Publish(...), you need someone to await bus.Subscribe<T>() accordingly. Underneath the covers, Rebus will use the .NET type as the topic, so if you
await bus.Subscribe<SomeEvent>();
in one application, and
await bus.Publish(new SomeEvent());
in another, your subscriber will receive the event.
TL;DR: You need to
await bus.Subscribe<StudentCreated>();
in the application where you want to receive published events of type StudentCreated.
Btw. you should EITHER use C#'s support for calling asynchronous methods
await bus.Send(yourMessage);
or invoke Rebus' synchronous API to do your work:
var syncBus = bus.Advances.SyncBus;
syncBus.Send(yourMessage); //< returns void
Rebus' synchronous methods do not deadlock the thread, e.g. if you're calling them from ASP.NET or WCF.

Parallel Handling Kafka messages in a Windows Service using C#

I have Windows service written in C#. Earlier we were using Event hubs with multiple partitions for message queuing. We recently moved to Kafka. For implementing Event hubs in c# , we have IEventProcessor.ProcessEventsAsync , which keeps listening to event hub notifications and is triggered whenever a message is posted to event hub , which runs asynchronously in the background
I did not find any equivalent method in Kafka.
My requirement here is to subscribe to a Kafka topic and continuously consume messages. When a message is consumed, some other operations are also supposed to executed for that message. For each message say the execution time takes around 15 mins, I want the Kafka consumer to consume all messages and keep it in queue as when it receives and writes it into a file. Other process should read the file, pick the message and do other operations. I want all of them to run simultaneously/parallelly.
PS : I have written a console application which can produce and consume one message.What I'm looking for is queuing and parallelism.

For paralellism Kafka implements what's known as consumer groups. Kafka stores the "offsets" (read: key of record across a topic) and also stores the offsets of where a given consumer group is also at in processing the records. This should allow you to create new consumer instances on the fly using the same program, and by changing the group allow two programs to consume the same data in paralell for different tasks.
I found this link helpful when I was creating my first consumer as well, in case you found a way to create it without a groupId: http://cloudurable.com/blog/kafka-tutorial-kafka-consumer/index.html
Hope this helps!

Have look at Silverback: https://silverback-messaging.net. It abstracts many of those concerns and the basic usage is as simple as this:
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services
.AddSilverback()
.WithConnectionToMessageBroker(options => options.AddKafka())
.AddKafkaEndpoints(
endpoints => endpoints
.Configure(
config =>
{
config.BootstrapServers = "localhost:9092";
})
.AddInbound(
endpoint => endpoint
.ConsumeFrom("my-topic")
.DeserializeJson(serializer => serializer.UseFixedType<SomeMessage>())
.Configure(
config =>
{
config.GroupId = "test-consumer-group";
config.AutoOffsetReset = AutoOffsetReset.Earliest;
})))
.AddSingletonSubscriber<MySubscriber>();
}
}
public class MySubscriber
{
public Task OnMessageReceived(SomeMessage message)
{
// TODO: process message
}
}

Implementing Publish Only Bus in MassTransit v3 with C# and RabbitMQ

I am trying to implement a publish only bus in MassTransit v3 with C# and RabbitMQ where the bus has no consumer. The concept is messages will be published and queued, then a separate microservice will consume messages from the queue. Looking at this SO answer, receive endpoints must be specified so that messages are actually queued. However this appears to contradict the common gotchas in the MassTransit docs, which states If you need to only send or publish messages, don’t create any receive endpoints.
Here is some sample code:
public class Program
{
static void Main(string[] args)
{
var bus = BusConfigurator.ConfigureBus();
bus.Start();
bus.Publish<IItemToQueue>(new ItemToQueue { Text = "Hello World" }).Wait();
Console.ReadKey();
bus.Stop();
}
}
public static class BusConfigurator
{
public static IBusControl ConfigureBus()
{
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), hst =>
{
hst.Username("guest");
hst.Password("guest");
});
cfg.ReceiveEndpoint(host, "queuename", e =>
{
e.Consumer<MyConsumer>();
});
});
return bus;
}
}
public interface IItemToQueue
{
string Text { get; set; }
}
public class ItemToQueue : IItemToQueue
{
public string Text { get; set; }
}
public class MyConsumer : IConsumer<IItemToQueue>
{
public async Task Consume(ConsumeContext<IItemToQueue> context)
{
await Console.Out.WriteLineAsync(context.Message.Text);
}
}
In this sample, I receive the message in the RabbitMQ queue as expected, and this is consumed by MyConsumer which writes Hello World to the console and the message is then removed from the Queue.
However, when I remove the following code from the above and re-run the sample:
cfg.ReceiveEndpoint(host, RabbitMqConstants.ValidationQueue, e =>
{
e.Consumer<MyConsumer>();
});
A temporary queue is created (with a generated name) and the message never seems to be placed into the temporary queue. This queue is then removed when the bus is stopped.
The problem I have is with a ReceiveEndpoint specified, the messages will be consumed and removed from the queue in the publisher program (meaning the consumer microservice wouldn't process queued items). Without a RecieveEndpoint specified, a temporary queue is used (and the consumer microservice would not know the name of this temporary queue), the message never seems to get queued and the queue is deleted when the bus is stopped which wouldn't be good if the program went down.
There is an example of a send only bus in the MassTransit docs but it is pretty basic so I was wondering if anyone had any suggestions?

The receive endpoint should be in your service, separate from the publish-only application. That way, the service will have the receive endpoint and consume the messages as they are published by the application.
If you have the receive endpoint in the application, the application will consume the messages since it has the same queue name specified in the receive endpoint.
What you need to do is create another service, with the same configuration (including the receive endpoint) - and take the receive endpoint out of your application. At that point, the service will have the receive endpoint and consume the messages from the queue. Even if the service is stopped, the messages will continue to be delivered to the queue and once the service is started they will begin consuming.

MassTransit - Update Messages to Client

I don't have very much experience using MSMQ and someone recommended I look at MassTransit to help implement a solution but I am having a hard time trying to figure out if using MassTransit + MSMQ is the right tool for the job.
We have a WPF application (3.5) that is used by multiple users. Persistence is done from the application (via NHibernate) to the database. Up until now, users would periodically refresh there view's in order to ensure they had the latest updates. However, we now want to send notification to each application instance when an entity is persisted using pub/sub messaging. The client applications are all run within the same domain and should be able to fulfill most dependencies required (e.g. installation of MSMQ on client machines).
To summarize: Client1 publishes an update message ---> ????? ----> All other active clients receive it.
As I am new to MSMQ, I'm not even sure what the architecture should look like.
Does each client machine need to have a local MSMQ queue to receive messages?
Do we just need to create a queue on a server and all clients listen for messages there? If so, will just a queue(s) suffice or do we need to create a service in order to distribute the messages correctly?
Is this even the right tool for the job?
I created a little POC hoping that it would work, but I ended up with what I think is termed "Competing Consumer". What I would like to happen is one application instance sends a message, and all application instances receive it.
Any suggestions, direction or advice would be greatly appreciated!
Here is the POC view model code (note - in my mind localhost would be replaced with a server that each app instance would send messages to):
Update: Added Network Key (kittens)
Update: I've uploaded the sample code https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0ByDMJXKmYB7zMjBmYzYwNDEtYzMwOC00Y2RhLTk1MDYtZjc0NTI2M2E3Y2Qy&hl=en_US
public class MainViewModel : IDisposable, INotifyPropertyChanged
{
private Guid id;
public MainViewModel()
{
id = Guid.NewGuid();
Publish = new RelayCommand(x => OnExecutePublishCommand(), x => !string.IsNullOrEmpty(Message));
Messages = new ObservableCollection<MessagePayload>();
Bus.Initialize(sbc =>
{
sbc.UseMsmq();
sbc.SetNetwork("Kittens");
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom(string.Format("msmq://localhost/{0}", ConfigurationManager.AppSettings["queue"]));
sbc.Subscribe(subs => subs.Handler<MessagePayload>(OnReceiveMessage));
});
}
public ICommand Publish { get; private set; }
private string message;
public string Message
{
get { return message; }
set
{
message = value;
SendPropertyChanged("Message");
}
}
public ObservableCollection<MessagePayload> Messages { get; private set; }
private void OnReceiveMessage(MessagePayload msg)
{
Application.Current.Dispatcher.Invoke(DispatcherPriority.Background,
new Action(() => Messages.Add(msg)));
}
private void OnExecutePublishCommand()
{
Bus.Instance.Publish(new MessagePayload{ Sender= id, Message = Message});
Message = null;
}
public event PropertyChangedEventHandler PropertyChanged;
private void SendPropertyChanged(string propertyName)
{
if (PropertyChanged != null)
{
PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
}
}
public void Dispose()
{
Bus.Instance.Dispose();
}
}
Update: Just in case anyone is interested we ended up splitting our "Event Bus" into two. For the server, we are using MassTransit. However, because Mass Transit requires "full profile" (.NET 4.0) and we wanted to stick with "client profile" for our WPF instances we are using SignalR for the client side event bus. An "observer" on the server event bus forwards messages to the client event bus.

All the machines on the same network can subscribe to given messages. They all need a local queue to read off of. Don't read off remote queues unless there's absolutely no other way.
What you described generally seems right. There's an message that gets published to all subscribers, they'll receive it and update their state. I have not worked with WPF in a while but generally how you're handling it seems acceptable. Note that it might take a little time to spin up the MT configuration, so you might want to do that on a background thread so you aren't blocking the UI.
Additionally, using the Multicast Subscription, you need to set a network key. It's automatically set to the machine name if not provided. You'll want to make sure they can talk to each other successfully.