I have been using WCF for a fair bit now, and I have come across several articles on MSDN regarding WCF's publisher subscriber model.
One of the requirements in a project I'm about to embark on requires me to have one server send out messages to multiple servers, thus, I have a few queries regarding the publisher/subscriber model
My primary concern with the this is:
What happens if 2 out of 5 of my subscribers are disconnected, due to say, PC rebooting. Do I have to manually handle the publisher re-sending the messages to the 2 subscribers when they reconnect back?
I need each message sent out by the publisher to positively reach all the other subscribers, if they are offline, the publisher has to be responsible to re-send the messages when the subscribers are back online. Because there can be a lot of messages being sent by the publisher, does that mean I will need some sort of queuing mechanism to store all the messages that are supposed to be sent to the offline subscribers, and re-send them when they are back online?
Is using WCF's publisher/subscriber mode fitting in my scenario? Are there any models out there that supports disconnected subscribers, automatically resending when the subscriber is back online, or do I have to custom code every single aspect to handle this?
You can try MSMQ, WCF supports it. The publisher will send the messages to 5 queues; each consumer will read its queue.
Related
I'm using rabbitmq in my microservices project and I saw these two methods
what are these and when do we use each
I suspect you're using something on top of RMQ, like EasyNetQ, because these aren't rabbit terms specifically, but in essence:
Publish publishes messages to a queue
Subscribe subscribes to a queue and defines the code that will act on the received message
A producer is a user application that sends messages while a consumer is a user application that receives messages.A queue is a buffer that stores those messages.
https://dev.to/mashaa/introduction-to-rabbitmq-49n8
In many pub/sub systems, publishers post messages to an intermediary message broker or event bus, and subscribers register subscriptions with that broker, letting the broker perform the filtering. The broker normally performs a store and forward function to route messages from publishers to subscribers. In addition, the broker may prioritize messages in a queue before routing.
https://en.m.wikipedia.org/wiki/Publish–subscribe_pattern
The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the producer doesn’t even know if a message will be delivered to any queue at all.
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I'd like to get your ideas as to how I can make my service process scale horizontally by being able to run it across multiple servers. It is a Windows service written in C#, and its purpose in life is to subscribe to our company's Exchange Web Service (EWS) so that it gets notified (via HTTP callback) whenever there's a new incoming email message. The service then gets the email message, processes it, sends a reply if possible, then goes back to sleep and waits for the next incoming email.
If I run it on more than one machine, I can either have all of them subscribing to EWS notification, or only one of them. If I have all of them subscribe, I am kind of hesitant because it might add burden to our MS Exchange infrastructure. Also this will result in all machines receiving and processing the email. I wouldn't want the sender to receive a reply N times (where N is the number of servers in the farm) for a given request message! Now if I have only one machine subscribing to EWS, that exposes me to a single point of failure.
I'd like to get your suggestions on how to address this. I'd love to have multiple servers process incoming messages by distributing email messages among them (perhaps I'll have to do this by making use of a message queueing server). Thanks.
Depends if you are scaling for reliability or throughput.
If reliability, you can have a primary and a standby process. The primary process subscribes and processes all emails. The standby process exchanges keep-alive messages with the primary and takes over as primary if the keep-alive times out.
If throughput, then a message queue mechanism , as you suggested, may be a good approach. You could run primary and standby as above, but the primary just pulls emails into a queue. A farm of message processors pulls off the queue.
Does SignalR replaces MSMQ or IMB MQ or Tibco message queues.
I have gone through SignalR.StockTicker
If we extend the functionality to read Stock tickers from multiple data sources and display to UI, will lit replaces usage of Message Queue
SignalR is designed for real-time messaging using several protocols (WebSockets, Long Polling, Server Sent Events, or Forever Frame). On the other hand, Service Bus type protocols (Such as MSMQ, RabbitMQ, Azure Service Bus) are designed for decoupled communication. The use cases can overlap, but in general, if you're looking for real-time updates (chat, tickers, notifications of a user status change) then SignalR is a good solution. One key difference is that MSMQ and the like do not require that the recipients, or subscribers, be online at the time the message is sent, whereas a SignalR client must be listening or it will miss the notification.
HTH
I have searched a lot, but still I had few doubts about MSMQ implementation of WCF service.
Hence I have put this quetion.
I want to implement single publisher and multiple subscribers asynchronous message system.
I have decided to use WCF service as the publisher.
I have multiple instances of the window service on different machines as the multiple subscribers.
Q. I want to know that, how this model can be implemented as the old MSMQ approach?
The object of following type would be used in the message
[Serializable]
public class Message
{
public string Signal{get;set;}
public Guid Identifier{get;set;}
}
In the above class, Identifier would be used by the windows service to decide whether the message was published for that service or not.
Q. How the different window services will read the same queue?
Q. Where the queue should be hosted?
Q. Is it possible to send the acknowledgement from Window service(subscriber) to WCF service(publisher)?
Any help would be appreciated.
In answer to your questions:
I want to know that, how this model can be implemented as the old
MSMQ approach?
First off, MSMQ does not support publish subscribe out of the box.
How the different window services will read the same queue?
So in publish subscribe, there in no ONE queue. Instead there are multiple queues, in fact one per participant in the pub sub scenario. So each publisher has a queue and each subscriber has a queue.
This configuration enables subscribers to send subscribe/unsubscribe messages to the publisher, and allows the publisher to send messages to the subscribers as necessary after evaluating the subscriptions.
Where the queue should be hosted?
These queues can be hosted locally to each participant, or can be hosted together in some clustered location.
Is it possible to send the acknowledgement from Window
service(subscriber) to WCF service(publisher)?
MSMQ does provide basic support for request/response messaging via response-queue and correlation-id message header fields, though this is not truly out of the box (as you are required to consume and program against these values)
If you are not required to use WCF there is a fairly mature platform called nservicebus which sits on top of MSMQ and does provide support for all the messaging patterns you need.
Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.