I'm new to SNS and SQS with MassTransit (usually I use RabbitMQ) and I'm having some trouble figuring out how to get it working properly. I'm building a scheduling service that broadcasts to all downstream services when any schedule is due, and then the downstream services decide whether or not they care.
My problem is that even though I have multiple consumers, only one SQS queue is created for the SNS topic representing the event. Once one of the consumers receives the message it triggers the visibility timeout and the other consumers never know about it.
In RabbitMQ one exchange could push a message to multiple queues. I might be getting confused by the AWS topology but shouldn't it be possible to create one queue for each consumer and prefix the service name (and probably also the scope)?
I would really appreciate any help with configuring MassTransit to create one queue for each consumer.
Here's my bus registration. I'm using Autofac for dependency injection.
builder.AddMassTransit(ctx =>
{
foreach (var type in consumers)
ctx.AddConsumer(type);
ctx.UsingAmazonSqs((context, configurator) =>
{
var lifetimeScopeFilter = context.GetService<LifetimeScopeFilter>();
var configuration = context.GetService<IAppSettings>();
configurator.UseFilter(lifetimeScopeFilter);
configurator.UseConsumeFilter(typeof(LoggingFilter<>), context);
configurator.UseConsumeFilter(typeof(SegmentFilter<>), context);
var host = configuration.Get("Bus:Host");
var scope = configuration.Get("Bus:Scope");
var secretKey = configuration.Get("Bus:SecretKey");
var accessKey = configuration.Get("Bus:AccessKey");
configurator.Host(host, hostConfig =>
{
hostConfig.Scope(scope);
hostConfig.SecretKey(secretKey);
hostConfig.AccessKey(accessKey);
hostConfig.EnableScopedTopics();
});
configurator.UseMessageRetry(r =>
{
r.Interval(5, TimeSpan.FromMilliseconds(500));
r.Handle<DBConcurrencyException>();
});
configurator.UseTransaction(x =>
{
x.Timeout = TimeSpan.FromSeconds(60);
x.IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted;
});
configurator.ConfigureEndpoints(context, new DefaultEndpointNameFormatter($"{scope}-", true));
});
});
Related
I am having a configuration problem with mass transit. It generates queues with MachineName_iisexpress... on AWS SQS.
I have the following config on startup:
services.AddMassTransit(busRegistrationConfiguration =>
{
busRegistrationConfiguration.AddConsumer<GenericConsumer>();
busRegistrationConfiguration.UsingAmazonSqs((context, cfg) =>
{
string awsRegion = Configuration.GetValue<string>("AWS-Region");
cfg.Host(awsRegion, host =>
{
host.AccessKey(Configuration.GetValue<string>("AWS-AccessID"));
host.SecretKey(Configuration.GetValue<string>("AWS-Secret"));
});
cfg.AutoStart = true;
cfg.ReceiveEndpoint("nxt-sqs-dev", x =>
{
x.UseMessageRetry(r => r.Interval(2, TimeSpan.FromMilliseconds(5000)));
x.ConfigureConsumeTopology = false;
x.Subscribe("fe-nxt-sqs-dev");
x.Consumer<GenericConsumer>(context);
});
//cfg.ConfigureEndpoints(context);
});
});
I did try to change the configuration but I couldn't solve my issue, did check the documentation etc.
Do not set cfg.AutoStart = true; unless you want MassTransit to start the bus endpoint (which uses a unique temporary queue name, in your case, MachineName_iisexpress_etc...). The bus endpoint is only used when you are using the request client from that bus instance. Remove that statement and MassTransit won't create the bus endpoint or its queue unless you actually use the request client from that bus instance.
Also, unrelated, but you should be configuring your consumer using:
x.ConfigureConsumer<GenericConsumer>(context)
I'm trying to build a Kafka consumer using MassTransit
I have this piece of code
var services = new ServiceCollection();
services.AddMassTransit(x =>
{
x.AddRider(rider =>
{
rider.AddProducer<string, Request>("request", m => m.Message.RequestId);
rider.UsingKafka((context, k) =>
{
k.Host("localhost:9092");
});
});
});
var provider = services.BuildServiceProvider();
var producer = provider.GetRequiredService<ITopicProducer<Request>>();
await producer.Produce(new Request()
{
RequestId = "abc123",
RequestedAt = DateTime.UtcNow
});
This is the simplest example of a producer from here
but when I try to run it, I get this exception
Unhandled exception. System.InvalidOperationException: No service for type 'MassTransit.Registration.IBusInstance' has been registered.
Looking at the example from their website, I see that it could be related to the fact that I haven't registered a RabbitMQ
x.UsingRabbitMq((context, cfg) => cfg.ConfigureEndpoints(context));
But I don't have a RabbitMQ, I only use Kafka in this scenario.
Is it necessary to register a bus with some other message broker in order to produce to Kafka?
From the documentation:
Riders, introduced with MassTransit v7, provide a new way to deliver messages from any source to a bus. Riders are configured along with a bus, and board the bus when it is started.
To add riders, there must be a bus instance. If you don't need a bus with a durable transport such as RabbitMQ, you can use the in-memory transport.
var services = new ServiceCollection();
services.AddMassTransit(x =>
{
x.UsingInMemory((context,cfg) => cfg.ConfigureEndpoints(context));
x.AddRider(rider =>
{
rider.AddProducer<string, Request>("request", m => m.Message.RequestId);
rider.UsingKafka((context, k) =>
{
k.Host("localhost:9092");
});
});
});
The bus needs to be started and stopped, which will also start/stop any riders on the bus. You can do this via IBusControl:
var provider = services.BuildServiceProvider();
var busControl = provider.GetRequiredService<IBusControl>();
await busControl.StartAsync(cancellationToken);
Or by adding the MassTransit Hosted Service if you're using the ASP.NET Core Generic Host.
services.AddMassTransitHostedService(); // in MassTransit.AspNetCore
I have a simple saga configuration in Asp.Net Core applicaiton:
services.AddSingleton<ISagaRepository<Request>, InMemorySagaRepository<Request>>();
services.AddMassTransit(x =>
{
x.AddSagaStateMachine<RequestStateMachine, Request>();
x.AddRequestClient<IRequestCreated>();
x.AddBus(provider => Bus.Factory.CreateUsingInMemory(cfg =>
{
cfg.UseInMemoryOutbox();
cfg.ConfigureEndpoints(provider);
}));
});
If later I send message to Saga over IRequestClient<IRequestCreated>:
var client = context.RequestServices.GetService<IRequestClient<IRequestCreated>>();
var response = await client.GetResponse<RequestCreatedResponse>(new
{
CorrelationId = Guid.NewGuid(),
ClientId = 1,
});
all works fine. But if I try same thing over IBus:
var mtbus = context.RequestServices.GetService<IBus>();
await mtbus.Send<IRequestCreated>(new
{
CorrelationId = Guid.NewGuid(),
ClientId = 1,
});
I get error A convention for the message type Sample.AspNetCore.Host.Saga.IRequestCreated was not found
What am I missing?
In your example above, because no service address is specified for the request client, it is publishing the request. Which gets routed to the consumer.
In your failing case, you're using Send, which without an address needs to know where to send the message. The only fallback is the convention, which isn't configured. If you want the same behavior, and honestly, since you're publishing a RequestCreated event, you should call Publish as well.
I'm using rebus with Autofac integration for webapi
Below is my configuration statement called on Application_Start of Global.asax.cs
When the app pool is restarted it doesn't recognise the subscriptions for events of a service listening to events published by the webapi, so the result is
DEBUG 09:56:12.842 dd2c1f8c-370d-4bd4-828c-27f033dfabdf RebusConfigurer+<>c.0 Sending Events.UserLoggedIn -> << no destinations>>
I need to restart the service for the webapi to receive subscriptions requests
I can see the configuration taken place with
DEBUG 09:55:50.744 890204c2-1dcb-4df0-9525-d6585d9692cb RebusAutofacConfig.25 Configuring rebus in queue :'xyz'
DEBUG 09:55:50.864 890204c2-1dcb-4df0-9525-d6585d9692cb RebusAutofacConfig.33 Configured rebus with 1 workers
Here is my webapi configuration to publish messages
public static class RebusAutofacConfig
{
private static readonly ILog Logger = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
internal static IContainer ConfigureRebus(this IContainer container)
{
var inputQueueName = System.Configuration.ConfigurationManager.AppSettings["queueName"];
if (inputQueueName.IsNullEmptyOrWhisteSpace())
{
inputQueueName = "xyz";
}
var errorQueue = inputQueueName + ".errors";
Logger.DebugFormat("Configuring rebus in queue :'{0}'", inputQueueName);
var bus = Configure
.With(new AutofacContainerAdapter(container))
.Serialization(s => s.UseJil(Jil.Options.IncludeInherited))
.Logging(l => l.Log4Net())
.Transport(t => t.UseMsmq(inputQueueName))
.Options(b => b.SimpleRetryStrategy(errorQueue, 5, true))
.Start();
Logger.DebugFormat("Configured rebus with {0} workers", bus.Advanced.Workers.Count);
return container;
}
}
Here is my windows service configuration to listen to messages
internal static IContainer ConfigureRebus(this IContainer container, ISettings settings)
{
Configure
.With(new AutofacContainerAdapter(container))
.Serialization(s => s.UseJil(Jil.Options.IncludeInherited))
.Logging(l => l.Log4Net())
.Transport(t => t.UseMsmq(settings.EventsQueueName))
.Routing(r => r.TypeBased().MapAssemblyOf<UserLoggedIn>(inputQueueName))
.Options(o => o.SimpleRetryStrategy(settings.EventsErrorQueueName, 5, true))
.Options(o =>
o.SpecifyOrderOfHandlers()
.First<UserCacheHandler>()
.Then<UserCreatedResetPasswordHandler>()
)
.Events(e =>
{
e.BeforeMessageHandled += (bus, headers,message, context, args) =>
{
System.Diagnostics.Trace.CorrelationManager.ActivityId = Guid.NewGuid();
};
})
.Start();
return container;
}
Seems like after the apppool is restarted no subscription is recognized.
I'm using a local Msmq.
Any ideas what can I be doing wrong?
From the configurations you posted, it looks like you are not storing your subscriptions anywhere.
This means that your Web API is storing subscription in memory, which explains why you always seem to have to start your processes in the
Web API
Windows Service
order.
Since MSMQ is a transport WITHOUT native pub/sub support (at least the way Rebus is using it), you need to configure a (persistent!) subscription storage for your scenario to work.
If you have e.g. a SQL Server available, I suggest you configure your Web API to use it to store subscriptions. It's done by importing the Rebus.SqlServer package, and then you go
.Subscriptions(s => s.StoreInSqlServer(...))
in the configuration spell.
You can read more about configuring a subscription storage on the wiki page about ISubscriptionStorage.
I am trying to create black box integration tests around some services that I have created using MassTransit. The gist of the service is that it receives a message on a MassTransit channel, processes the message and sends the message on a different MassTransit channel (note that this is not a Request-Reply semantic but more of a component in a pipeline):
IBusControl bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost"),
h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ReceiveEndpoint(host, "Queue1", ep =>
{
ep.Handler<ItemStarted>(context =>
{
ItemFinished item = FinishMessage(context.Message);
context.Publish(item);
});
});
});
bus.Start();
In order to test this I think what I need is a synchronous way to receive messages from the bus. Is there any way to do this:
IBusControl bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
var host = cfg.Host(new Uri("rabbitmq://localhost"),
h =>
{
h.Username("guest");
h.Password("guest");
}));
bus.Start();
bus.Publish(new ItemStarted());
// This is what does not seem to exist
ItemFinished finished = bus.Receive<ItemFinished>(timeout : Timespan.FromSeconds(5));
// Assertions about finished
Is there a way to receive messages directly from MassTransit without wiring up a consumer class or lambda?
There is a MultiTestConsumer that can be used to receive and track messages that are produced during testing. You can see the class here:
https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit/Testing/MultiTestConsumer.cs
And you can see how it is used here:
https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit.Tests/MultiTestConsumer_Specs.cs
It can also be used as a regular subscribed consumer, which sets up bindings in RabbitMQ to receive published messages.
I would recommend checking out the TestFramework, as there are a bunch of asynchronous testing helpers in there.