I'm trying to handle request authorization in a microservice based architecture using a message queue (RabbitMQ).
I've got a receiver and sender configured fine as a console application in .NET Core per these instructions. However, when using this in a real world example, my application receiving project isn't collecting messages as a consumer.
I'm assuming I have to register the consumer in the Startup.cs, but I can't seem to get this working.
My consumer/responder code:
public class RabbitMqHandler
{
private readonly IJWTFactory _jwtFactory;
public RabbitMqHandler(IJWTFactory jWTFactory)
{
_jwtFactory = jWTFactory;
}
public void Register()
{
var mqFactory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = mqFactory.CreateConnection())
{
Console.WriteLine("Listening on Rabbit MQ");
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "Authorize", durable: false, exclusive: false, autoDelete: false, arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var jwtToken = Encoding.UTF8.GetString(body);
Console.WriteLine("Rceived Message");
var validatedToken = _jwtFactory.ValidateTokenSignature(jwtToken);
SendResponse(validatedToken);
};
channel.BasicConsume(queue: "Authorize", autoAck: true, consumer: consumer);
}
}
}
public void Deregister()
{
}
Startup.cs to register the
.AddSingleton()
Edit: I've added some additional listening code, this is definitely running on startup, but RabbitMQ is not showing the app as a consumer or a channel:
public static class ApplicationBuilderExtentions
{
public static RabbitMqHandler Listener { get; set; }
public static IApplicationBuilder UseRabbitListener(this IApplicationBuilder app)
{
Listener = app.ApplicationServices.GetService<RabbitMqHandler>();
var life = app.ApplicationServices.GetService<IApplicationLifetime>();
life.ApplicationStarted.Register(OnStarted);
//press Ctrl+C to reproduce if your app runs in Kestrel as a console app
life.ApplicationStopping.Register(OnStopping);
return app;
}
private static void OnStarted()
{
Listener.Register();
}
private static void OnStopping()
{
Listener.Deregister();
}
}
To summarise:
How do I correctly configure a consumer in .NET Core to consume messages?
Is this just the wrong to expect a Message Queue to manage request/response style communication?
Should I just be using an API call to authenticate and authorize users?
Answer as provided by #Evk (in the comments):
"using is designed to dispose things, it calls Dispose when you reach the end of the using block. BasicConsume is not a blocking call, so it starts consumption and returns immediately.
Right after that the end of the using blocks is reached for both channel and connection, disposing them (and disposing them is the same as closing)."
I'd like to add the following:
You can quickly try if removing the using brings the desired result very easily. Change the following code line:
using (var connection = mqFactory.CreateConnection())
to:
var connection = mqFactory.CreateConnection();
This will instantly do the trick. But be aware it also removes proper disposal - so you need to add that - here is an article from Microsoft describing how IDisposable is to be implemented correctly.
Related
I am trying to create a simple application that will publish some messages to a topic with MQTT (library I am using is M2Mqtt.net) and then I want to subscribe to the topic once the messages have already been sent and then have them all be received and then discarded, because they have been received.
I am using mosquitto 2.0.12 as the broker
This is the publisher:
public class MessagePublisher : IMessagePublisher
{
private readonly MqttClient _client;
public MessagePublisher()
{
_client = new MqttClient("localhost");
// clean session needs to be set to false so that it retains all the missed messages, not just the last one
_client.Connect(Guid.NewGuid().ToString(), "username", "password", false, byte.MaxValue);
}
public void Publish(string topic, string message, bool retain = false)
{
Console.Write($"Sent: {topic}, {message}");
_client.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, retain);
Total.SentAndReceived.Add(message, null);
}
}
This is the listener:
public class MessageReceiver : IMessageReceiver
{
private readonly MqttClient _client;
public MessageReceiver()
{
_client = new MqttClient("localhost");
}
public void Subscribe(params string[] topics)
{
_client.Subscribe(topics, new[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
_client.MqttMsgPublishReceived += client_receivedMessage;
}
public void Connect()
{
// clean session needs to be set to false so that it retains all the missed messages, not just the last one
_client.Connect(Guid.NewGuid().ToString(), "username", "password", false, byte.MaxValue);
}
public void Disconnect()
{
_client.Disconnect();
}
static void client_receivedMessage(object sender, MqttMsgPublishEventArgs e)
{
var message = Encoding.Default.GetString(e.Message);
Console.WriteLine($"Message Received: {message}");
if (Total.SentAndReceived.ContainsKey(message))
Total.SentAndReceived[message] = message;
}
}
And this is the main application:
public static class Program
{
public static void Main(string[] args)
{
var messageReceiver = new MessageReceiver();
var publisher = new MessagePublisher();
for (var i = 1; i <= 10000; i++)
{
publisher.Publish("Devices/", i.ToString(), true);
}
messageReceiver.Subscribe("Devices/");
messageReceiver.Connect();
Thread.Sleep(5000);
var b = Total.SentAndReceived.Where(x => x.Value == null);
Console.WriteLine($"{b.Count()} Missed Messages");
}
}
The problem I am having is that there are missed messages. And the number of missed messages always changes when I run the application. And it's not that last n messages being missed it's the first n messages.
I hope that if I was to build a service that would listen to the published messages. If the services stops for any reason. Once the service comes back online, the messages sent in that downtime would be received.
I think you have a misunderstanding around some terms here.
First, MQTT does not generally queue messages. The only time the broker will queue messages is if the receiving client has already already been connected and subscribed to the topic at QOS > 0. If that client then disconnects before the publisher sends the messages the broker will queue the messages. They will then only be sent to the receiving client if they then reconnect with the same client id and have the clean session flag set to false. This is the only way that messages will be queued up.
Since you appear to be using randomly generated client ids (Guid.NewGuid().ToString()) this will not work. You also appear to be trying to subscribe before you connect, again that won't work.
Secondly, retained messages have nothing to do with message queuing as described above. A message is retained if the retained flag is set at the point of publishing. The broker will then store that specific message and deliver it ever time a client subscribes to the matching topic. This message will be sent before any other messages on the topic. If another message with the retained flag is published it will replace the previous message, there can only be 1 retained message per topic.
I created a small demo to show the RabbitMQ basics. Unfortunatly it doesn't work as expected an has two issues. I am using .NET Core 3.1 and RabbitMQ.Client 6.2.2
I created the Employee class which receives messages from the task queue. The first employee is working nice but if I start more employees they don't work (don't receive messages). And I can't figure out why that would be.
And if I have a lot of messages in the queue (before starting the second employee) I see that all messages in the tasks queue get ACKed when the second starts and then after a short time they become UNACKed again. Somehow weird.
But mainly: why do the other employees not work?
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System;
using System.Text;
using System.Threading;
namespace DemoTasks.Employee
{
class Employee
{
static void Main(string[] args)
{
string clientName = "Employee-" + Guid.NewGuid().ToString();
Console.Title = clientName;
Console.WriteLine("Moin moin");
IConnectionFactory connectionFactory = new ConnectionFactory
{
HostName = "localhost",
Port = 5672,
VirtualHost = "/",
UserName = "user",
Password = "password",
ClientProvidedName = clientName
};
using (IConnection connection = connectionFactory.CreateConnection(clientName))
{
using (IModel model = connection.CreateModel())
{
model.ExchangeDeclare("jobs", "fanout", false, false, null);
model.QueueDeclare("tasks", true, false, false);
model.QueueBind("tasks", "jobs", "", null);
EventingBasicConsumer consumer = new EventingBasicConsumer(model);
consumer.Received += OnWorkReceived;
model.BasicConsume("tasks", false, clientName + ":OnWorkReceived", consumer);
Console.ReadLine();
model.Close();
}
connection.Close();
}
Console.WriteLine("Wochenende ... woooh !!!");
}
private static void OnWorkReceived(object sender, BasicDeliverEventArgs e)
{
EventingBasicConsumer consumer = (EventingBasicConsumer)sender;
IModel model = consumer.Model;
string task = Encoding.UTF8.GetString(e.Body.ToArray());
Console.Write("working on: " + task + " ... ");
Thread.Sleep(5000);
Console.WriteLine("done!");
model.BasicAck(e.DeliveryTag, false);
}
}
}
I think your problem is about setting PrefetchCount on your channel. It's about how many messages that one consumer can get from rabbit and cache them on itself to process them.
If don't set it, one consumer can consume all messages on queue and no time to get messages by other consumers, so you can set it by using channel.basicQos(1) or basicqos(0,1,false). By this setting every consumer can get one message after send ack to rabbit then can get another one.
When set prefetch count to lower number, can affect on performance because your consumer must ask rabbit more to get messages .
For detail information see this: https://www.rabbitmq.com/consumer-prefetch.html
Up till now for the past 3 months, I still have 0 clue how SignalR works at the JIT (Just-in-time) level. I'm trying to build a Hub that sends data to the client just in time, and the client will then receive the data and work along with it.
EDIT: Incase you have no idea what I mean by JIT Sending and
Receiving,
I meant it by the server being able to send connected socket clients data when there is new data available. The socket connection will only be closed either when the server is shutdown/has an issue OR the client disconnects from the socket. So in short, no matter what, when new data arises from the server, it will always send that data ONE BY ONE to connection clients.
So here's what I'm missing out/confused about:
Is the SubscribeToAll (Check out TickerHub.cs below) Method the place where I call when I have new data to notify and beep to the clients or where is it?
I know how the asynchronous WriteToChannel works. Basically it sends a collection, item by item to the client. Key issue is, how do I convert this entire function to JIT? And where do I handle the list of clients subscribed to this hub?
Currently, TickerHub.cs keeps retrieving a dataset (named CurrencyPairs) and then broadcasts it to the clients indefinitely. I have a background service that syncs and updates the CurrencyPairs 24/7. I just need a SignalR expert's help to explain/show how I can invoke the Hub from the background service and then allow the hub to broadcast that new data to the connected clients.
TickerHub.cs
public class TickerHub : Hub, ITickerHubClient
{
private IEnumerable<CurrencyPair> _currencyPairs;
private readonly ICurrencyPairService _cpService;
public TickerHub(ICurrencyPairService cpService)
{
_cpService = cpService;
}
public async Task<NozomiResult<CurrencyPair>> Tickers(IEnumerable<CurrencyPair> currencyPairs = null)
{
var nozRes = new NozomiResult<CurrencyPair>()
{
Success = true,
ResultType = NozomiResultType.Success,
Data = currencyPairs
};
return nozRes;
}
// We can use this to return a payload
public async Task<ChannelReader<NozomiResult<CurrencyPair>>> SubscribeToAll()
{
// Initialize an unbounded channel
//
// Unbounded Channels have no boundaries, allowing the server/client to transmit
// limitless amounts of payload. Bounded channels have limits and will tend to
// drop the clients after awhile.
var channel = Channel.CreateUnbounded<NozomiResult<CurrencyPair>>();
_ = WriteToChannel(channel.Writer); // Write all Currency Pairs to the channel
// Return the reader
return channel.Reader;
// This is a nested method, allowing us to write repeated methods
// with the same semantic conventions while maintaining conformity.
async Task WriteToChannel(ChannelWriter<NozomiResult<CurrencyPair>> writer)
{
// Pull in the latest data
_currencyPairs = _cpService.GetAllActive();
// Iterate them currency pairs
foreach (var cPair in _currencyPairs)
{
// Write one by one, and the client receives them one by one as well
await writer.WriteAsync(new NozomiResult<CurrencyPair>()
{
Success = (cPair != null),
ResultType = (cPair != null) ? NozomiResultType.Success : NozomiResultType.Failed,
Data = new[] {cPair}
});
}
// Beep the client, telling them you're done
writer.Complete();
}
}
}
In case you want to find out if my client sided code doesn't work well, here it is
using Microsoft.AspNetCore.SignalR.Client;
using Newtonsoft.Json;
using Nozomi.Client.Data.Interfaces;
using Nozomi.Data;
using Nozomi.Data.CurrencyModels;
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Nozomi.Client
{
public class NozomiClient
{
private CancellationToken _tickerStreamCancellationToken;
private string ServerPath;
private HubConnection _hubConnection;
public NozomiClient(string serverPath)
{
ServerPath = serverPath;
_hubConnection = new HubConnectionBuilder()
.WithUrl(serverPath)
.Build();
}
public async Task InitializeAsync()
{
await _hubConnection.StartAsync();
}
public async Task StreamTickers()
{
// Setup the channel for streaming
var streamTickerChannel = await _hubConnection.StreamAsChannelAsync<NozomiResult<CurrencyPair>>("SubscribeToAll", CancellationToken.None);
// Setup the asynchronous data stream
// https://learn.microsoft.com/en-us/aspnet/core/signalr/streaming?view=aspnetcore-2.1#net-client
//while (await streamTickerChannel.WaitToReadAsync())
//{
// while (streamTickerChannel.TryRead(out var cp))
// {
// Console.WriteLine(JsonConvert.SerializeObject(cp));
// }
//}
_hubConnection.On<CurrencyPair>("SubscribeToAll", cp =>
{
Console.WriteLine(cp);
});
while (!_tickerStreamCancellationToken.IsCancellationRequested)
{
if (await streamTickerChannel.WaitToReadAsync())
{
while (streamTickerChannel.TryRead(out var cp))
{
Console.WriteLine(JsonConvert.SerializeObject(cp));
}
}
Console.WriteLine("Processing");
Thread.Sleep(1000);
}
}
public ICurrencyPair CurrencyPairs { get; }
public ISource Sources { get; }
}
}
I am new to RabbitMQ and am having trouble finding a VS2017 C# example that does more that prints to the Console. I can send and receive messages no problem, but I would like to take the contents of the message and actually use it. Below is the code I have:
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.IO;
namespace MyCode
{
class Program
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "xxx.xx.x.x", UserName = "MyTest", Password = "MyTest", Port = 5672 };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "MyTest", durable: false, exclusive: false, autoDelete: false, arguments: null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
DoSomething(message);
};
channel.BasicConsume(queue: "MyTest", autoAck: true, consumer: consumer);
}
}
}
static void DoSomething(string message)
{
File.AppendAllText(#"C:\RMQ.txt", message);
}
}
}
The problem is, I can't ever seem to get anything out of the Consumer.Received step. Any help would be much appreciated!
EDIT::
I was able to make it work by using this code instead:
var factory = new ConnectionFactory() { HostName = "xxx.xx.x.x", UserName = "MyTest", Password = "MyTest", Port = 5672 };
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "MyTest", durable: false, exclusive: false, autoDelete: false, arguments: null);
BasicGetResult consumer = channel.BasicGet("MyTest", true);
if (consumer != null)
{
string message = Encoding.UTF8.GetString(consumer.Body);
DoSomething(message);
}
}
}
Anybody see any issues with this?
The problem with your first piece of code is that your program finishes execution before it can handle any messages. EventingBasedConsumer is asynchronous and won't actually prevent you program from exiting. You need to implement a wait of some sort to be able to actually handle messages. Try adding Thread.Sleep(10000); just after Channel.BasicConsume and check if there are any messages being processed.
For fire and forget applications, that tries to setup the read, try not to use the "using" statement for your connection and channel. They will go out of scope (set for garbage collection) once the the subroutine finish the execution.
Obviously, you need to have the console application up and running to receive the message. So, I did put a Console.Readline() in the calling app to prevent the thread from exiting.
We are using RabbitMQ for queuing messages in C# .Net (EasyNetQ Client).
i want one consumer app (C# Console App) listen to one queue and provide multiple handlers for each message type.
I implemented this scenario and my code is here :
using (var advancedBus = RabbitHutch.CreateBus("host=localhost;prefetchcount=100")
.Advanced)
{
var queue = advancedBus.QueueDeclare("MyQueue");
advancedBus.Consume(queue, x => x
.Add<MessageType1>((message, info) =>
{
Console.WriteLine("MessageType1 Body : " + message.Body.Body);
})
.Add<MessageType2>((message, info) =>
{
Console.WriteLine(" MessageType2 Body: " + message.Body.Body);
}).ThrowOnNoMatchingHandler = false);
}
My Problem :
But when i execute this consumer it does nothing. do not any thing happen.
i publish messages to that queue like this :
using (var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced)
{
var queue = advancedBus.QueueDeclare("MyQueue");
if (advancedBus.IsConnected)
advancedBus.Publish(Exchange.GetDefault(), queue.Name, false, false,
new Message<MessageType1>(change));
else
result = false;
}
What is the problem.
Ok, after testing this code, these are the problems:
First of all, you're disposing your advancedBus right after you register for consumption. You need to remember that when you invoke IAdvanceBus.Consume, you're only registering a callback for each message. If you dispose the bus immediately after registration, your delegate can't be invoked since the connection was already closed. So, you'll to remove the using statement around the rabbit declaration (don't forget to dispose it when you're done):
var advancedBus = RabbitHutch.CreateBus("host=localhost;prefetchcount=100").Advanced
Second, the immediate flag has been deprecated and shouldn't be used, the message doesn't seem to be getting to the queue. Change Publish to:
advancedBus.Publish(Exchange.GetDefault(), queue.Name, true, false,
new Message<MessageType1>(change));
Also, if you're running this from a console application, don't forget to use Console.ReadKey so your main thread doesn't terminate.
Here's a working code sample:
static void Main()
{
var change = new MessageType1();
var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced;
ConsumeMessage(advancedBus);
var queue = advancedBus.QueueDeclare("MyQueue");
if (advancedBus.IsConnected)
{
advancedBus.Publish(Exchange.GetDefault(), queue.Name, true, false,
new Message<MessageType1>(change));
}
else
{
Console.WriteLine("Can't connect");
}
Console.ReadKey();
}
private static void ConsumeMessage(IAdvancedBus advancedBus)
{
var queue = advancedBus.QueueDeclare("MyQueue");
advancedBus.Consume(queue, registration =>
{
registration.Add<MessageType1>((message, info) =>
{
Console.WriteLine("Body: {0}", message.Body);
});
});
}