I have multiple microservices. They communicate through message broker as currently, I am using Masstransit rabbitmq. Things are working fine.
Now, I have a case where one service A publishes a message and another service B consumes it. Service B sends a number of emails which takes some time. If it fails, it invokes a new event where I want to send a negative acknowledgment and want service A to republish again after some delay. I know I can through an exception in service B consume method but this is not possible here. Send email runs in an independent way. How can I republish the message from service A to B?
The answer to your question has nothing to do with MassTransit, or any other library, which you can use to support the flow. You need to design the message flow according to your needs.
It is not very clear, why sending an email takes some time, and why it can fail, so the following is purely based on assumptions.
If service B sends a batch of emails by consuming a single message, it is not transactional. If it sends 100 emails and fails on 101, what would happen? If you have a message retry policy, it will send the first 100 emails again.
I would definitely try to limit a single message scope to handle a single transaction. Split each email send operation to its own message and handle those (potentially thousands) of messages, it's totally fine. At least, in case of a retry, it will retry one operation only. Each individual operation is confirmed, not the whole batch. It is fine for service A to command service B to send a number of emails. It is the responsibility of service B to split the number of emails it needs to send to a number of individual messages. Service B can perfectly send messages to itself.
How you handle errors is your own decision. For all the cases when I had to deal with sending emails, it was like this:
Each email is a separate message
If it fails, I check what the reason is. Usually, you don't get a failure from the mail provider immediately, unless the email address is obviously incorrect, or the provider itself is down.
Prevent sending a message to wrong emails in the first place, you don't need to overload the system with such obvious errors.
If the email provider is down, engage the message retry policy accordingly. Ensure to have a circuit breaker in place to manage the backpressure from the provider if you hammer it too hard and get rate-limited. You can also rate-limit your own consumers. All of those are built-in middleware in MassTransit.
Only retry transient errors. If the error is permanent, there's no need to retry.
If all the retries suddenly fail, you can still engage the redelivery. It's built-in to MassTransit. Still, ask yourself a question why can it even happen?
Sending a NACK message from service A to service B is trivial. If the error happens, instead of throwing, publish or send another message for service A. I don't see any issue with that.
You can always check if your retry or redelivery count is depleting by looking at the consumer context metadata. You can decide to only send a NACK message to service A if all the retries have failed.
There's nothing wrong with throwing per se, as you get the retry and redelivery middleware there to help you handling transient failures. Service A can always listen to Fault events if those policies exhaust their retry counts and the message will land in the poison queue. The poison queue is valuable as it allows you to analyse the nature of the failiure.
Again, it's not a technical issue. You need to decide how you want to handle this workflow. MassTransit would help you to implement it in the way you want it, but you first need to decide what you want.
Related
I am completely new to Azure and I have a function which is triggered by a service bus queue, the message is a JSON string. Now I want to check if after 3 minutes of receiving the message another arrived.
how do I achieve this?
You may want to use the concept of "Timers in Durable Functions".
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-timers?tabs=csharp
Are you using "azure functions" or do you want to have a function in azure e.g. in a webservice? I am not so sure ;)
In short no! You would have to poll the servicebus message queue yourself to control that behaviour.
https://stackoverflow.com/a/48899686/12040634
What is the scenario that would require you to check for a subsequent message after 3 minutes?
I have a function which is triggered by a service bus queue… I want to check if after 3 minutes of receiving the message another arrived.
Azure Service bus can schedule or defer a message. The two are not the same. Deferring means you need to keep a message and retrieve it at some point yourself. Scheduling, requires creating a new message. But, the two could be combined. Upon receiving of the message (message A), it could be deferred and a new message scheduled (message B), with its payload being the sequence number of the message A. When the message’s B time comes, it will be received by your code and you can request the deferred message B.
This is rather straightforward but you’ll need a few changes to the default Function behaviour. You’ll need to specify that your function needs to be injected with the receiver to be able to defer the incoming message A, and create a sender to post a scheduled message. Also, the function code will now have to distinguish between “data” messages (A) and control messages (B). And that all will only work with the in-process Functions SDK as the newer, Isolated Worker SDK doesn’t support Service Bus SDK types.
But, there are alternatives. You could use Durable Functions to build your mini workflow. Or you could use one of the middleware abstractions for messaging to simplify the implementation with function. MassTransit and NServiceBus with SendLocal()would do.
And good luck with conquering Azure. There are many ways to tackle any problem. Especially with cloud providers.
Problem
Currently, our website is setup so that when an action is taken that requires an email to be sent, our website will make a call the the SMTP server to attempt to send an email. The problem lies when the SMTP server goes down for whatever reason. We don't store any outgoing emails in any fashion so if the email to be sent fails, it's is lost forever (not really as it can easily be regenerated, but we don't have a mechanics to let us know it failed, except Azure Application Insights). While we also have the website send the devs an email when exceptions occur, for obvious reasons, we will not receive those emails.
Goal
Our goals is to stop having our website send an email directly to the email relay server. Instead, implement a solution that would send emails and have the ability to recover should a problem occur.
Stop the website from sending the emails
Ability to recover from transient or side issues/exceptions
Log as much activity as possible concerning the email (send attempts/fails/etc)
Ability to recover the activity logs from potential transient or side issues/exceptions
Ability to re-trigger a email to be sent if ever necessary (optional)
Solution
I read a 3-part article that sounds like it would solve this issue and I'm currently developing it.
I'm building a process using Microsoft.Azure.ServiceBus Topics and Subscriptions to manage sending emails from our website. I've gone through many samples and have been successfully able to SendAsync() a Message, ReceiveAsync() a Message, and CompleteAsync() or AbandonAsync() it appropriately.
Side-note: I'm now exploring how to work with the RetryPolicy to see if this will help me defer a retry to a bit of a longer period, though I'm not sure if I can/should use it for that.
While most of the process has been built so far, so I can understand the underlying infrastructure, I'm also still within the planning phase to make sure we plan appropriately.
We are currently trying to figure out the best or most appropriate workflow for this process. We figured two Topics would be needed: one for the emails to be sent EmailTopic, and one for the logs to record LogTopic.
The reason for the LogTopic is to handle any transient issues when attempting to save a log activity to the database. For example: I successfully retrieve an email to be sent. I then attempt the send the email and log this attempt. The email gets sent successfully. I then try to log this activity, but the database just went down and I'd would not be able to log this activity. The second Topic should alleviate that, but what happens if that goes down?
Here's our current workflow:
Website inserts data into a database that defines the email to be sent (currently, we'll have a field for the Body which will be the email contents itself, another table to hold the Email Templates which will contain the contents around the Body field, along with from, to, CC, BCC, and file attachments)
Website sends a small Message to the EmailTopic with the MessageId of the inserted record
A Stateless Service Fabric Service listens for messages
Receive the Message, get all details from the database for the record
Build the SMTPClient and attempt to send the email to the SMTP server
Send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken (attempt to send email)
If successful, CompleteAsync() the Message and send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken: "email sent"
If unsuccessful, AbandonAsync() the Message and send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken: "email failed to send" (after 10 attempts message would automatically be placed in the DeadLetterQueue
In this workflow, the LogTopic will contain all the actions taken and will be stored in the database when the message(es) is(are) received. Obviously, if messages are abandoned for any reason and sent the the DeadLetterQueue, we will have a process to try to insert them at a later point.
Questions
We thought about just storing logs to the database within the workflow, but the question "what if the db goes down in the meantime?" (hence when Azure Central US went down last week) came up so we decided to use this 2nd Topic. Obviously, if the Service Bus is down, we can't send this message and I don't know how to recover from that, except log ETWs and check them some other way. Should I be attempting a DB save first, and if that fails, send a Message to the Topic?
Are there too many things going on in this service and should I split some operations around?
Is there a flaws or missing items in the workflow itself that we aren't taking into consideration?
Should we be using 1 Topic and add a Label to the message so we know it's a log vs email to be sent? Maybe using filters (not sure how to properly do this or if it's appropriate for this workflow yet)?
Are we asking too many questions in this 1 SO post and should split each question apart?
I think the work flow can be improved in such a way that the failures can be tracked and resolvable. I am providing my solution in achieving this.
Service Bus Topics supports multiple subscribers for single sender. This is achievable with the help of Subscriptions.
Instead of sending the message to two Topics, You can create two Subscriptions under a Topic. Refer here for filtering messages into Subscriptions using Rules.
You can create different Rules for the Subscriptions. Once the message is sent to a Topic, the custom properties of the message gets validated against the Rules of each Subscription. Based on the result of the validation, the message will get into the Subscription which has the required Rule condition.
Lets say the Subscriptions created are Email and Log. The Stateless Service Fabric Service should listen for the messages form both these Subscriptions.
Website should send the message to the Topic with the custom properties for Email Subscription.
Whenever a message is received by the Stateless Service Fabric Service, it should start a thread to send the mail.
Once the email is sent successfully, a Success message should be sent to the Topic with the custom properties for Log Subscription. In case of failure in sending mail, the message in Email Subscription should be dead lettered and a Failure message should be sent to the Topic with appropriate custom properties for Log Subscription.
The Stateless Service Fabric Service which listens also for the messages from Log Subscription will create a thread to write to the database as and when a message is arrived. In case of failure in writing into database, the message in Log Subscription should be dead lettered.
You can monitor the count of the dead letter messages of both the Subscriptions to ensure there is no failure. If the count is greater than 0, there should be manual intervention in Resubmitting the dead letter messages to the Topic. There are may tools available in market to monitor and resubmit the dead letter messages or you can also develop a custom application for doing that.
I guess this workflow should work as expected. Once this is installed, the only thing that need to be taken care is the dead letter messages in both the Subscriptions.
What is the best way to implement in service bus messages that are requiring once a week or once a day etc.
I am thinking of having a separate windows service that just drops in messages from the database into the service bus but is there another way?
In simple terms i want a message that once it is processed, it will appear again in the queue in a specified amount of time to be processed again.Obviously once i process a message i can tell service bus to delete the message or appear again in the queue.
You will need to have some external process (e.g. your windows service) which sends the message in the first place, on schedule. You can use Azure Scheduler to do that, see http://www.prasadthinks.com/blog/2015/07/11/azure-scheduler-can-post-to-azure-service-bus-queue-and-topic/
When you are processing your message, you can do what you are describing i.e. re-send a copy of the message, using BrokeredMessage.ScheduledEnqueueTimeUtc property so that it arrives at the time you want. But I wouldn't do that, does not feel right. If you have your external processing already sending messages on schedule, just rely on that 100%.
I am fixing a .net app written on top of nServiceBus and have a few questions:
The app is configured AsA_Publisher and when it starts it waits for incoming
connections on a socket, do you know why it might have been implemented like so?
Why sockets? This socket is created during the Run method of a class which implements class IWantToRunAtStartup.
Once a message arrives, the message is written to a queue (Q1). The message
is then read from the queue(Q1). The format of the message is changed and then
inserted into yet another queue (Q2). The message is then read from the queue
(Q2) and sent to another application by calling a web service. The whole idea is
to change the message format and send it off to the final destination. If
nServiceBus is built on top of MSMQ, then why is the application creating more
queues and managing them?
I see absolutely nothing about Publish or Subscribe anywhere in the project. I guess it is relying on the socket to receive messages and if so then it is not really taking advantage of nServiceBus's queuing facility? Or am I lost...
If queues are needed and if I was to build this I will have one app writing to
the queue (Q1), another app reading from the queue (Q1) and changing the format
and inserting to another queue (Q2) and finally a third app reading from the
(Q2) and sending it off to the web service. What do you think?
Thanks,
I see nothing wrong with opening a socket in Run in an IWantToRunAtStartup. It must somehow be required that the service can be reached through some custom protocol implemented on top of sockets.
Processing the incoming socket messages by immediately bus.Sending a message is also the way to go - the greatest degree of reliability is achieved by immediately doing the safest thing possible: sending a durable message.
Performing the message translation in a handler and bus.Sending the result in another message is ALSO the way to go - especially if the translation is somehow expensive and it makes sense to be able to pick up processing at this point if e.g. the web service call fails.
Making a web service call in a message handler is also be the way to go - especially if the web service call is idempotent, so it doesn't break anything if the message ever gets retried.
In other words, it sounds like the service correctly bridges a socket-based interface to a web service-based interface.
It sounds weird, however, that the service employs multiple queues to achieve this. With NServiceBus it would be entirely sufficient with one single queue: the service's input queue.
Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.