I'm trying to notify the user when a message is not received after some time, using MassTransit and RabbitMQ.
From what I read, the timeout is set using the TimeToLive property when the message is published. When that specified time runs out, the message should be automatically added to a Dead Letter queue, named with a "_skipped" at the end.
How do I retrieve messages from Dead Letter queues? In my attempt below, the message is added to the both queues right away, and it never times out.
I think I could do this using sagas, but it seems like a over complicated solution for such a simple problem, so I would like to avoid using it if at all possible.
static void Main(string[] args)
{
var bus = CreateBus("rabbitmq://localhost/", "guest", "guest", true);
var msg = new TestMessage("First Message");
LogMessageSent(msg);
bus.Publish(msg, c => c.TimeToLive = TimeSpan.FromSeconds(15));
Console.ReadKey();
bus.Stop();
bus = CreateBus("rabbitmq://localhost/", "guest", "guest", false);
msg = new TestMessage("SecondMessage");
LogMessageSent(msg);
bus.Publish(msg, c => c.TimeToLive = TimeSpan.FromSeconds(15));
Console.ReadKey();
bus.Stop();
}
private static IBusControl CreateBus(string rabbitUrl, string username, string password, bool enableEndpoint)
{
var bus = Bus.Factory.CreateUsingRabbitMq(c =>
{
var host = c.Host(new Uri(rabbitUrl), h =>
{
h.Username(username);
h.Password(password);
});
if (enableEndpoint)
{
c.ReceiveEndpoint(host, "TestQueue", x =>
{
x.Handler<TestMessage>(e => LogMessageReceived(e.Message, "TestQueue"));
});
}
c.ReceiveEndpoint(host, "TestQueue_skipped", x =>
{
x.Handler<TestMessage>(e => LogMessageReceived(e.Message, "TestQueue_skipped"));
});
});
bus.Start();
return bus;
}
private static void LogMessageSent(TestMessage msg)
{
Console.WriteLine(string.Format("{0} - Message \"{1}\" sent.", DateTime.Now.ToString("HH:mm:ss"), msg.Content));
}
private static Task LogMessageReceived(TestMessage msg, string queueName)
{
Console.WriteLine(string.Format("{0} - Message \"{1}\" received on queue \"{2}\".", DateTime.Now.ToString("HH:mm:ss"), msg.Content, queueName));
return Task.CompletedTask;
}
public class TestMessage
{
public string Content { get; }
public TestMessage(string content)
{
Content = content;
}
}
Because you are calling Publish, the message is sent to every subscriber. Since each receive endpoint is adding the consumer, that creates a subscription (and subsequent exchange binding in RabbitMQ) for that message type. You can disable this by specifying BindMessageExchanges = false on that skipped receive endpoint. You will need to manually remove the exchange binding on the broker.
As to your TimeToLive question, that isn't how it works. TimeToLive is passed to the broker, and if the message expires, it is moved to a broker-specified dead-letter queue, if so configured. It is not moved to the skipped queue which has a different meaning in MassTransit. In MassTransit, the skipped queue is for messages that are delivered to a receive endpoint but there wasn't a consumer configured on that endpoint to consume the message.
For RabbitMQ, you can configure the dead-letter queue in MassTransit by using:
endpoint.BindDeadLetterQueue("dead-letter-queue-name");
This will configure the broker so that messages which reach their TTL are moved to the specified exchange/queue. Then your consumer on that receive endpoint will be able to consume them (again, be sure to set BindMessageExchanges = false on the dead-letter receive endpoint.
For example:
c.ReceiveEndpoint(host, "TestQueue_expired", x =>
{
x.BindMessageExchanges = false;
x.Handler<TestMessage>(e => LogMessageReceived(e.Message, "TestQueue_expired"));
});
And then your original receive endpoint:
c.ReceiveEndpoint(host, "TestQueue", x =>
{
x.BindDeadLetterQueue("TestQueue_expired");
x.Handler<TestMessage>(e => LogMessageReceived(e.Message, "TestQueue"));
});
Related
I can't find a way to confirm that a message sent to the Kafka server has been received correctly by the server.
I need a confirmation that the message was sent and received correctly in order to act on the C # code in case the message was not received by the server.
I have read that there is a property called ack to configure that, but I can't find how to receive such confirmation.
You'd use a DeliveryReport handler
Refer documentation - https://docs.confluent.io/clients-confluent-kafka-dotnet/current/overview.html
Async
var t = producer.ProduceAsync("topic", new Message<Null, string> { Value="hello world" });
t.ContinueWith(task => {
if (task.IsFaulted)
{
...
}
else
{
...
Console.WriteLine($"Wrote to offset: {task.Result.Offset}");
}
});
Synchronous
public static void handler(DeliveryReport<Null, string>)
{
...
}
public static process(...)
{
...
producer.Produce(
"my-topic", new Message<Null, string> { Value = "hello world" }, handler);
}
This worked for me:
bool messagePublished;
var result = await _producer.ProduceAsync(yourTopic, yourMessage);
if (result.Status == PersistenceStatus.Persisted)
messagePublished = true;
Notice that you have multiple PersistenceStatus values.
You can find more info on the official confluent documentation.
I'm trying to send a Scheduled Message using the UseDelayedExchangeMessageScheduler along with the rabbitmq_delayed_message_exchange plugin. I do setup the bus like this:
public void StartUpBus()
{
_bus = Bus.Factory.CreateUsingRabbitMq(ConfigureBus);
_bus.Start();
}
private void ConfigureBus(IRabbitMqBusFactoryConfigurator busConfigurator)
{
var host = busConfigurator.Host(new Uri(_connectionInfo.ConnectionString), h =>
{
h.Username(_connectionInfo.User);
h.Password(_connectionInfo.Password);
});
busConfigurator.UseDelayedExchangeMessageScheduler();
busConfigurator.ReceiveEndpoint(host, "schedule-send-endpoint.inbox", endpoint => {
endpoint.PrefetchCount = 1;
endpoint.Consumer( () => new AScheduledConsumer() );
});
}
The consumer is simple as this
public class AScheduledConsumer : IConsumer<AScheduledMessage>
{
public Task Consume(ConsumeContext<AScheduledMessage> context)
{
return Console.Out.WriteLineAsync($"Message received at {DateTime.Now}");
}
}
Here is the message I'd like to send
// Simple role interface to easily identify bus travelling data
public interface IMessage
{
}
public class AScheduledMessage : IMessage
{
}
Then I try to send the message with something like this
var message = new AScheduledMessage();
var delay = TimeSpan.FromMinutes(1);
Uri destinationUri = new Uri("rabbitmq://localhost/schedule-send-endpoint.inbox");
await Console.Out.WriteLineAsync($"Message sent at {DateTime.Now}");
return _bus.ScheduleSend(
destinationUri,
delay,
message
);
I expected to have the message delivered about one minute after the send, but the message ends in the _skipped queue. Here after the message as it is dumped by the "Get Message" function in the RabbitMq management ui.
The server reported 0 messages remaining.
Exchange schedule-send-endpoint.inbox_skipped
Routing Key
Redelivered ○
Properties
message_id: d7040000-4392-98e7-c8e1-08d7e3d61e54
correlation_id: d7040000-4392-98e7-cc70-08d7e3d61e49
delivery_mode: 2
headers:
Content-Type: application/vnd.masstransit+json
MT-Host-Assembly: Infrastructure.Messaging.RabbitMq.Test.ConsoleApp
MT-Host-AssemblyVersion: 1.0.0.0
MT-Host-FrameworkVersion: 4.0.30319.42000
MT-Host-MachineName: GABROS-NB
MT-Host-MassTransitVersion: 5.2.1.1808
MT-Host-OperatingSystemVersion: Microsoft Windows NT 6.2.9200.0
MT-Host-ProcessId: 26984
MT-Host-ProcessName: Infrastructure.Messaging.RabbitMq.Test.ConsoleApp.vshost
MT-Reason: dead-letter
infrastructure.correlation-id: 029ea5c6-e5ee-44b7-8851-84d3b6ebd191
infrastructure.user-id: anonymous
publishId: 1
content_type: application/vnd.masstransit+json
Payload
1649 bytes
Encoding: string
{
"messageId": "d7040000-4392-98e7-c8e1-08d7e3d61e54",
"correlationId": "d7040000-4392-98e7-cc70-08d7e3d61e49",
"conversationId": "d7040000-4392-98e7-29ed-08d7e3d61e5d",
"sourceAddress": "rabbitmq://localhost/bus-GABROS-NB-Infrastructure.Messaging.RabbitMq.Test.ConsoleApp.vshost-4hnyyynd1kcqqhmibdm68io7fu?durable=false&autodelete=true",
"destinationAddress": "rabbitmq://localhost/schedule-send-endpoint.inbox",
"messageType": [
"urn:message:MassTransit.Scheduling:ScheduleMessage[[Infrastructure.Messaging.Test:AScheduledMessage]]",
"urn:message:MassTransit.Scheduling:ScheduleMessage"
],
"message": {
"correlationId": "d7040000-4392-98e7-cc70-08d7e3d61e49",
"scheduledTime": "2020-04-18T20:21:47.1796308Z",
"payloadType": [
"urn:message:Infrastructure.Messaging.Test:AScheduledMessage",
"urn:message:Infrastructure.Messaging:IMessage"
],
"destination": "rabbitmq://localhost/schedule-send-endpoint.inbox",
"payload": {}
},
"sentTime": "2020-04-18T20:21:46.8178828Z",
"headers": { },
"host": {
"machineName": "GABROS-NB",
"processName": "Infrastructure.Messaging.RabbitMq.Test.ConsoleApp.vshost",
"processId": 26984,
"assembly": "Infrastructure.Messaging.RabbitMq.Test.ConsoleApp",
"assemblyVersion": "1.0.0.0",
"frameworkVersion": "4.0.30319.42000",
"massTransitVersion": "5.2.1.1808",
"operatingSystemVersion": "Microsoft Windows NT 6.2.9200.0"
}
}
And these are the Exchanges I find in the RabbitMQ
Name Type Features Message rate in Message rate out+/-
(AMQP default) direct D
Infrastructure.Messaging.Test:AScheduledMessage fanout D
amq.direct direct D
amq.fanout fanout D
amq.headers headers D
amq.match headers D
amq.rabbitmq.trace topic D I
amq.topic topic D
bus-GABROS-NB-Infrastructure.Messaging.RabbitMq.Test.ConsoleApp.vshost-4hnyyynd1kcqqysnbdm6jy77ny fanout AD
schedule-send-endpoint.inbox fanout D 0.00/s 0.00/s
schedule-send-endpoint.inbox_skipped fanout D 0.00/s 0.00/s
What I find strange is the total absence of an x-delay header in the message and no x-delayed-message Exchange created in rabbitMq, as if the UseDelayedExchangeMessageScheduler was totally ignored ... I think I'm doing something wrong, but really can't find the culprit !
--- UPDATE ---
As pointed out by #ChrisPatterson, the message must be sent from a MessageScheduler.
While still didn't find a solution for MassTransit 5.2.1, this code works using MassTransit v6.4.2 and dotnet core 2:
var message = new AScheduledMessage();
var delay = TimeSpan.FromMinutes(1);
Uri destinationUri = new Uri("rabbitmq://localhost/schedule-send-endpoint.inbox");
var ms = new MessageScheduler(new DelayedExchangeScheduleMessageProvider(bus, bus.Topology as IRabbitMqHostTopology))
await Console.Out.WriteLineAsync($"Message sent at {DateTime.Now}");
return ms.ScheduleSend(
destinationUri,
delay,
message
);
If you are scheduling messages from the bus, and not inside a consumer, you need to use a message scheduler class. What you see happening in your example above is that it is sending to the destinationAddress as if that was Quartz, but it isn't. To use the delayed exchange outside of a ConsumeContext you have to create a MessageScheduler class, and pass it the RabbitMQ Delayed Exchange Scheduler Provider.
It isn't obvious, I will update the documentation and try to make it easier for scheduling from the bus using non-Quartz schedulers.
I've also added a new method, CreateRabbitMqMessageScheduler that is an extension method on IBus that basically creates the required components to schedule using the delayed exchange:
if (bus.Topology is IRabbitMqHostTopology topology)
return new MessageScheduler(new DelayedExchangeScheduleMessageProvider(bus, topology));
That is what's done under the hood.
When I add a new user to my IdentiyServerService the following MassTransit code is called:
var newUserCreated = new UserCreated
{
UserId = userId.ToString(),
Name = user.Name
};
await _bus.Publish(newUserCreated);
My destination is that my ProfileService receive this event by RabbitMq.
My RabbitMq configuration in my Startup.cs (IdentiyServerService)
private static void ConfigureBus(ContainerBuilder builder)
{
builder.Register(context =>
{
return Bus.Factory.CreateUsingRabbitMq(config =>
{
var host = config.Host(new Uri("rabbitmq://localhost"), h =>
{
h.Username("guest");
h.Password("guest");
});
});
}).As<IBus, IBusControl, IPublishEndpoint>().SingleInstance();
}
I start the bus like this
//Startup.cs IdentityServerService
var container = containerBuilder.Build();
var busControl = container.Resolve<IBusControl>();
busControl.Start();
The configuration in my ProfileService look quite almost the same. The difference is, that I add an consumer in my Startup.cs (ProfileService)
config.ReceiveEndpoint(host, "user_queue", ep =>
{
ep.Consumer<UserCreatedConsumer>(); // The consumer is registered explicitly this time.
});
I add an IConsumer as well
public class UserCreatedConsumer : IConsumer<UserCreated>
{
public Task Consume(ConsumeContext<UserCreated> context)
{
var user = context.Message;
Debug.WriteLine("My debug string here");
return TaskUtil.Completed;
}
}
When I create a new user the messages receive the RabbitMq (publish rates increase). But then nothing goes on. The total number of messages in the Queued messages does not change.
I have two connections (I would expect IdentityServerService and ProfileService) and I have different queues (I was only expecting one: user_queue)
When I implement the IConsumer inside my IdentityServerService I receive the message.
I have no error log, warnings or something else.
Anyway... 1) Why does ProfileService not receive the message? 2) And why do I have so much queues?
If you need some more information... please tell
Edit
When I send a message in the rabbitMq-management my ProfileService receive a message, but now I get the following error
MassTransit.Messages Error: 0 : R-FAULT
rabbitmq://localhost/user_queue_new Value cannot be null.
Parameter name: source,
System.Runtime.Serialization.SerializationException: An exception occurred while deserializing the message envelope ---> System.ArgumentNullException: Value cannot be null.
Parameter name: source
We're currently switching out NServiceBus for MassTransit and I'm having a little difficulty with the request/response pattern.
In NServiceBus, I'm able to do reply in the Handler and it goes back to the client that sent it.
In MassTransit, it appears as though the response is being sent back to the queue that it was consumed from, thus creating a loop...
Weird thing, if I'm creating the Bus using InMemory, and both client and consumer on the same machine, I do not have the issue.
I am expecting my client to catch the response, but instead my Consumer picks it up, which is also odd, since it's not setup to receive that message type...
Am I missing something in the client's Request setup?
Client:
....
IRequestClient<IWorklistRequest, IWorklistResponse> client = CreateRequestClient(busControl, WorklistEndpointUri);
Console.Write("Sending Request");
Task.Run(async () =>
{
IWorklistRequest request = new WorklistRequest
{
CurrentDateFrom = new DateTime(2016, 11, 07)
};
var response = await client.Request(request);
Console.WriteLine("Worklist Items retrieved: {0}", response.ExamItemList.Length);
}).Wait();
....
static IRequestClient<IWorklistRequest, IWorklistResponse> CreateRequestClient(IBusControl busControl, string endpointAddress)
{
Console.WriteLine("Creating Request client...");
var serviceAddress = new Uri(endpointAddress);
IRequestClient<IWorklistRequest, IWorklistResponse> client =
busControl.CreateRequestClient<IWorklistRequest, IWorklistResponse>(serviceAddress, TimeSpan.FromSeconds(10));
return client;
}
Consumer:
public Task Consume(ConsumeContext<IWorklistRequest> context)
{
_log.InfoFormat("Received Worklist Request with Id: {0}", context.RequestId);
try
{
var result = _provider.GetAllWorklistsByStartDate(context.Message.CurrentDateFrom);
IWorklistResponse response = new WorklistResponse
{
ExamItemList = result.ToArray()
};
// the below is sending the response right back to the original queue and is getting picked up again by this same consumer
context.Respond(response);
}
catch (Exception ex)
{
_log.Info(ex.Message);
}
return Task.FromResult(0);
}
If you are using RabbitMQ, and you are using the request client, you should not see this behavior.
There is a sample that demonstrates how to use the request client on the MassTransit GitHub repository: https://github.com/MassTransit/Sample-RequestResponse
The code above appears to be correct, and the Respond() call should use the response address from the request message, which is a direct endpoint send to the temporary bus address.
There is pretty extensive unit test coverage around this area, and the sample above was updated and verified with the latest version of MassTransit. You might consider deleting/recreating your RabbitMQ virtual host and running your application from scratch (start the response service first, so that the endpoints are setup).
I'm monitoring the topic and subs and messages are getting in, however my masstransit consumer is not receiving anything.
Here's how it's been setup:
var bus = Bus.Factory.CreateUsingAzureServiceBus(
cfg =>
{
var azSbHost = cfg.Host(new Uri(CloudConfigurationManager.GetSetting("ServiceBus.Url"))
, host =>
{
host.TokenProvider = TokenProvider
.CreateSharedAccessSignatureTokenProvider
(CloudConfigurationManager.GetSetting("ServiceBus.SharedAccessKeyName"),
CloudConfigurationManager.GetSetting("ServiceBus.AccessKey"),
TokenScope.Namespace);
});
cfg.ReceiveEndpoint(azSbHost,
e =>
{
e.Consumer<PingConsumer>();
});
//azSbHost.
});
The Ping Consumer:
public class PingConsumer : IConsumer<Ping>
{
public async Task Consume(ConsumeContext<Ping> pingContext)
{
pingContext.Respond(new Pong
{
Message = "Pong: " + pingContext.Message.Message
});
}
}
And the sender:
var pong = await _bus.CreatePublishRequestClient<Ping, Pong>(TimeSpan.FromSeconds(10),null ).Request(
new Ping {Message = "Ping: " + message});
In Azure, I'm seeing my message count climbing up and not going down. So messages are getting to the queue, but consumer is not consuming the message.
I was missing a VERY important key call to make it all work on both client and server side.
Bus.Start