MSMQ concurrent processing design issue - c#

I have a windows service written in C# that reads from MSMQ and based on the type of the message it assigns them to Agents that process that message in a worker thread. The application starts with no agents and are created dynamically at runtime as messages arrive in the MSMQ
Here is a basic figure of how it works:
If the agent worker thread is busy doing work the message is queued to its local queue. So far so good. But if for some reason if the service is stopped, the local queue content is lost.
I am trying to figure out what could be the best way to handle this scenario. Right now the local queues are a System.Concurrent.ConcurrentQueue. I could probably use a Sql Ce db or some other persistent storage, but i am worried about performance. The other thing in my mind is to read from MSMQ only when agents are ready to process message, but the problem is that I don't know what message the MSMQ will contain.
What possible approaches can I take on this issue?

Your design is basically implements the following pattern: http://www.eaipatterns.com/MessageDispatcher.html
However, rather than using actual messaging you are choosing to implement the dispatcher in multithreaded code.
Rather, each processing agent should be an autonomous process with it's own physical message queue. This is what will provide message durability in case of failure. It also allows you to scale simply by hosting more instances of the processing agent.

I have built a similar system dependent on Redis. The idea is that it provides memory - fast data access isolated from the rest of the application, and will not shut down when my service does. Furthermore, it will eventually persist my data to the disk, so I get a good compromise between reliability and speed.
If you designed it so that each client read from its own message queue that would be hosted in Redis, you could keep the queue independent from the service's downtime, and each worker's load apportioned when you next start the service.

Why don't you simply create two new msms queues to receive the messages for Agenta and agentb, and create a new agent that ( transactionally ) fetch the command from the main queue and dispatch the message to the proper agent queue ?

Related

WCF and MSMQ - Push messages from Queue to Services?

I just want to find out if this is possible or whether I'm over complicating it.
I have a web application (Let's call it central) that needs to interact with a WCF service that's running on multiple workstations. i.e. The user will select the workstations to send messages to and the web application will need to do a call to each of the workstations. These workstations won't be online all the time and in the worst case there can be just over 600 workstations that messages will need to be sent to at a single time.
I'm thinking of having a separate WCF service running on the central machine that will function as a sort of "proxy" between the web app (central) and the workstations. The web app will then make a single call to this service with a list of messages, the service will then process this list and add the messages to a queue.
From what I've read so far, the workstations will need to poll this central queue for messages but this seems like it will increase overhead quite a bit. Is it possible to push the messages down to the workstation as they are added to the queue?
I've never used MSMQ before and I'm fairly new to WCF as well so if there's a simpler way of achieving this do tell.
I'm thinking of having a separate WCF service running on the central machine that will function as a sort of "proxy" between the web app (central) and the workstations
I don't see any problem with you using an additional service to act as a "proxy".
the workstations will need to poll this central queue for messages
I would probably advise that you have a central "inbox" of messages to process on the central server. The "proxy" then locally processes the inbox. Have the web site send the jobs to the proxy.
Push model
For the workstations, rather than having them read from the server, have the server send the messages to the workstations. Have a unique queue for each workstation and that queue be situated on the workstation itself. Have the service send the message targeted for the workstation to the workstation's queue. As each message from the central queue/inbox is processed by the proxy, the proxy removes said message from the queue and moves onto the next one.
MSMQ Performance
Generally in MSMQ it's better to do network writes than network reads for scaling reasons. A multitude of computers all reading from the same queue is hurtful to performance. In other words, have the "central" service write to the remote queues (MSMQ will take care of transmitting the message when the machine is available).
the workstations will need to poll this central queue for messages but this seems like it will increase overhead quite a bit
Correct. Use BeginRead. Polling is hurtful to CPU and/or a waste of a good thread.

Communication between Processing and C# app?

I would like to pass an object from a C# application to be processed? How may I accomplish this task?
Create a socket and pass the data over TCP using the Network library
You might want to implement a queue of some sort. Reasons:
This ensures that your objects can queue up while processing occurs.
If the processor is down, you don't lose any submitted objects as they will wait in the queue.
Decouples your C# application and your processing service (or whatever it may be)
The type of queue you need depends on your environment. Here are two good options:
MSMQ and how to interface with it in C# for local or network-based applications
Windows Azure Queue or Service Bus if you working in the Azure cloud

nServiceBus used with sockets

I am fixing a .net app written on top of nServiceBus and have a few questions:
The app is configured AsA_Publisher and when it starts it waits for incoming
connections on a socket, do you know why it might have been implemented like so?
Why sockets? This socket is created during the Run method of a class which implements class IWantToRunAtStartup.
Once a message arrives, the message is written to a queue (Q1). The message
is then read from the queue(Q1). The format of the message is changed and then
inserted into yet another queue (Q2). The message is then read from the queue
(Q2) and sent to another application by calling a web service. The whole idea is
to change the message format and send it off to the final destination. If
nServiceBus is built on top of MSMQ, then why is the application creating more
queues and managing them?
I see absolutely nothing about Publish or Subscribe anywhere in the project. I guess it is relying on the socket to receive messages and if so then it is not really taking advantage of nServiceBus's queuing facility? Or am I lost...
If queues are needed and if I was to build this I will have one app writing to
the queue (Q1), another app reading from the queue (Q1) and changing the format
and inserting to another queue (Q2) and finally a third app reading from the
(Q2) and sending it off to the web service. What do you think?
Thanks,
I see nothing wrong with opening a socket in Run in an IWantToRunAtStartup. It must somehow be required that the service can be reached through some custom protocol implemented on top of sockets.
Processing the incoming socket messages by immediately bus.Sending a message is also the way to go - the greatest degree of reliability is achieved by immediately doing the safest thing possible: sending a durable message.
Performing the message translation in a handler and bus.Sending the result in another message is ALSO the way to go - especially if the translation is somehow expensive and it makes sense to be able to pick up processing at this point if e.g. the web service call fails.
Making a web service call in a message handler is also be the way to go - especially if the web service call is idempotent, so it doesn't break anything if the message ever gets retried.
In other words, it sounds like the service correctly bridges a socket-based interface to a web service-based interface.
It sounds weird, however, that the service employs multiple queues to achieve this. With NServiceBus it would be entirely sufficient with one single queue: the service's input queue.

Selective Reading From a Queue--Custom MSMQ Service, ESB, or Something Else?

Looking for some ideas/pattern to solve a design problem for a system I will be starting work on soon. There is no question that I will need to use some sort of messaging (probably MSMQ) to communicate between certain areas of the system. I don't want to reinvent the wheel, but at the same time I want to make sure I am using the right tool for the job. I have been tinkering with and reading up on NServiceBus, and I'm very impressed with what it does--however I'm not sure it's intended for what I'm trying to achieve.
Here is a (hopefully) very simple and conceptual description of what the system needs to do:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
For each client, messages must be handled in order (FIFO). However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time. For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve. So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
My current thinking is that I really just need to use vanilla MSMQ, create a service to accept messages and write them to one or more queues, then create a process to read messages from the queue(s) and handle/process them. However, the reliability, auditing, scaleability, and ease of configuration you get with something like NServicebus looks very appealing.
Is an ESB the wrong tool for the job? Is there some other technology or pattern I should be looking at?
Update
A few clarifications.
Regarding processing messages "in order"--in the context of a single client, the messages absolutely need to be processed in the order they are received. It's complicated to explain the exact reasons why, but this is a firm requirement. I neglected to mention that only one message per client would ever be processed concurrently. So even if there were 10 worker threads and only one client had messages waiting to be processed, only one of those messages would be processed at a time--there would be no worry of a race condition.
I believe this is generally possible with vanilla MSMQ--that you can have a list of messages in a queue and always take the oldest one first.
I also wanted to clarify a use case for the round robin ordering. In this example, I have two clients (A and B) who send messages, and only one worker thread. All queues are empty. Client A has lost connectivity overnight, so at 8am sends 1000 messages to the service. These messages get queued up and the worker thread takes the oldest one and starts processing it. As this first message is being processed, client B sends a message into the service, which gets queued up (as noted, probably in a separate queue). When Client A's first message completes processing, the logic should check whether client B has a message (it's client B's "turn"), and since it finds one, process it next.
If client B hadn't sent a message during that time, the worker would continue processing client A's messages one at a time, always checking after processing to see if other client queues contained waiting messages to ensure that no client was being starved.
Where I still feel there may be a mismatch between an ESB and this problem is that an ESB is designed to facilitate communication between services; what I am trying to achieve is a combination of messaging/communication and a selective queuing system.
So the system will either have to be
able to query the queue for a specific client,
Searching through an MSMQ queue for a message from a particular client using cursors can be inefficient and doesn't scale.
or create separate queues per client (automatically, since the
clients won't be known at compile time) and pull from them in rotation.
MSMQ cannot create queues automatically. All messages have to be sent to a known queue first. Your own custom dispatcher service, though, could then create new queues on demand and put copies of the messages in them.
[[I avoid saying "move" messages as you can't do that with application code; you can only read a message and create a new message using the original data. This distinction is important when you are using Source Journaling, for example.]]
Cheers
John Breakwell
Using an ESB like NServiceBus seems like a good solution to your problem. But based on your conceptual description, there's some things to consider. Let's go through your requirements step-by-step, using NServiceBus as a possible ESB solution:
I have a service that clients can send messages to. The service is "Fire and Forget"--the most the client would get back is something that may say success or failure (success being that the message was received).
This is easily done with NServiceBus. You can Bus.Send(Message) from the client. If your client requires an answer, you can use Bus.Return(ErrorCode). You mention that "success being that the message was received". If you use an ESB like NServiceBus, it's up to the messaging platform the deliver the message. So, if your Bus.Send doesn't throw an exception, you can be sure that the message has been sent properly. Because of this you don't probably have to send success / failure messages back to the client.
The handling/processing of each message is non-trivial, and may take up significant system resources. For this reason only X messages can be handled concurrently, where X is a configurable value (based on system specs, etc.). Incoming messages will be stored in queue until it's "their turn" to be handled.
When using NServiceBus, you can configure the the number of worker threads by setting the "NumberOfWorkerThreads" option. If your server has multiple cores / cpus, you can use this setting to balance the work load.
For each client, messages must be handled in order (FIFO).
This is something that may cause problems depending on your requirements. ESBs in general don't promise to process the messages in-order, if they have many threads working on the messages. In a case of NServiceBus, you can send an array of messages from the client into the bus and these will be processed in-order. Also, you can solve some of the in-order messaging problems by using Sagas.
However, some clients may send many messages in succession (thousands or more), for example if they lost connectivity for a period of time
When using an ESB solution, your server doesn't have to be up for the client to work. Clients can still send messages and the server will start processing them as soon as it's back online. Here's a small introduction on this.
For this reason, messages must be handled in a round-robin fashion across clients--no client is allowed to gorge and no client is allowed to starve.
This isn't a problem because you've decided to use messages :)
So the system will either have to be able to query the queue for a specific client, or create separate queues per client (automatically, since the clients won't be known at compile time) and pull from them in rotation.
Could you expand on this? I'm not sure of your design on this one.

Is this the correct approach to poll the database?

I am creating a WCF service (CALLER) for Azure. The service(CALLER) calls async methods of another third party service(EXTN). The third party service calls the callback methods of another WCF service (LISTNER) hosted by me on Azure. CALLER enter the service details in the databsae with status = PENDING.
In the callback service (LISTNER) I am updating the status of the request as COMPLETED/FAILED in the database.
But I want the CALLER should be notified when status is updated in the SQL Azure db.
I am thinking of creating a worker thread which will poll the database periodically to check the status update and notify the CALLER about this.
Is there any other better / efficient alternative to this approach?
The features you're looking for are implemented in the AppFabric service bus.
Not really. There is another way (not sure it works on azure) by using a the integrated SQL message queueing (queue on updates via trigger), and your thread could continously poll then (there is a way to have a the read WAIT for an etnry in teh queue, so you issue one and it waits), but besides that...
...no, not from the database level.
I have a similar application and I handle it by a ntification trigger OUTSIDE The database (i.e. notifications are sent from the business logic that values change).
Another option is to use Queues and have the caller poll for notification messages from the listener. The Service Bus can be used, by having the Caller subscribe to event notifications sent from the Listener. In your scenario though it doesn't provide much more than the Queues do - if you are behind the firewall, the Service Bus uses polling as well.
Queues are probably the most efficient way to send notifications - that's why they were created in the first place. The Service Bus is used to create semi-permanent connections between different services by providing a lot more features than simple message passing. That makes it a bit less flexible, requires a bit more programming. Its billing model (charge per SB connection) reflect this too. You are not expected to use a lot of SB connections.

Categories