I have inherited an application that pulls messages out of an MSMQ does some processing to them and then adds some data to database depending on what is in the message. The messages are getting pushed into the Queue by a third party application I do not control.
I do not know much about MSMQ, although I do have a basic understanding of how to use the APIs.
Anyway, I have noticed that the messages never get deleted, our client definately never explictly deletes them, and I can look in computer management and see the messages back to when the server was last rebooted.
Is this wrong? Will the messages start to automatically get deleted when the queue reaches some maximum size or will they just pile up there forever slowly taking up more memory?
Once a message has been processed, it is normal practice to remove it from a queue (transactionally or otherwise).
I'd suspect that while this isn't best practice, the queue is cleared on reboot, and as long as there's a sufficient amount of resources available, you'll never actually run into a problem.
That said, I'd opt for setting something up to periodically clean up the queue so you don't overwhelm the server. I'm not too familiar with MSMQ, but is there some way that you can tell if a message has been processed? Even if it's an additional service that runs, checks the messages in the queue and sees if they already appear in the database, and deletes them if they do? That way, you wouldn't need to modify the codebase you inherited, since it's working properly as-is.
Once you decide on a solution, please post an update here - I'm interested to know how you end up dealing with this problem. Thanks!
"Anyway, I have noticed that the messages never get deleted, our client definately never explictly deletes them, and I can look in computer management and see the messages back to when the server was last rebooted."
Sounds like the messages are Express if there are none around from before the last reboot. Express messages are only stored in RAM and not persisted to disk so restarting the MSMQ service will destroy them. This is probably why the volume of messages has never reached a critical level.
As MSMQ uses kernel memory and disk space for memory storage, eventually one of the two would give out and cause you server stablity issues so your plan to have a cleanup process is a good one.
Cheers,
John Breakwell (MSFT)
Related
We are running multiple instances of a windows service that reads messages from a Topic, runs a report, then converts the results into a PDF and emails them to a user. In case of exceptions we simply log the exception and move on.
The use case we want to handle is when the service is shut down we want to preserve the jobs that are currently running so they can be reprocessed by another instance of the service or when the service is restarted.
Is there a way of requeueing a message? The hacky solution would be to just republish the message from the consuming service, but there must be another way.
When incoming messages are processed, their data is put in an internal queue structure (not a message queue) and processed in batches of parallel threads, so the IbmMq transaction stuff seems hard to implement. Is that what I should be using though?
Your requirement seems to be hard to implement if you don't get rid of the "internal queue structure (not a message queue)" if this is not based on a transaction oriented middleware. The MQ queue / topic works well for multi-threaded consumers, so it is not apparent what you gain from this intermediate step of moving the data to just another queue. If you start your transaction with consuming the message from MQ, you can have it being rolled back when something goes wrong.
If I understood your use case correctly, you can use Durable subscriptions:
Durable subscriptions continue to exist when a subscribing application's connection to the queue manager is closed.
The details are explained in DEFINE SUB (create a durable subscription). Example:
DEFINE QLOCAL(THE.REPORTING.QUEUE) REPLACE DEFPSIST(YES)
DEFINE TOPIC(THE.REPORTING.TOPIC) REPLACE +
TOPICSTR('/Path/To/My/Interesting/Thing') DEFPSIST(YES) DURSUB(YES)
DEFINE SUB(THE.REPORTING.SUB) REPLACE +
TOPICOBJ(THE.REPORTING.TOPIC) DEST(THE.REPORTING.QUEUE)
Your service instances can consume now from THE.REPORTING.QUEUE.
While I readily admit that my knowledge is shaky, from what I understood from IBM’s [sketchy, inadequate, obtuse] documentation there really is no good built in solution. With transactions the Queue Manager assumes all is well unless it receives a roll back request and when it does it rolls back to a syncpoint, so if you’re trying to roll back to one message but two other messages have completed in the meantime it will roll back all three.
We ended up coding our own solution updating the way we’re logging messages and marking them as completed in the DB. Then on both startup and shutdown we find the uncompleted messages and programmatically publish them back to the queue, limiting the DB search by machine name so if we have multiple instances of the service running they won’t duplicate message processing.
We use rabbit mq to send messages to a server for processing.
We require the server to ack a message. That way if the server happens to die whilst processing the message, we will retry the message when it restarts, or with a different server.
The problem is, on a very rare occasion, we will get a message that deterministically crashes the server. This is because we call into some open source native dlls, those dlls have bugs, and sometimes these dlls just cause the process to crash with no exception. Of course it would be ideal to fix those bugs, but we don't expect to fix all such issues in pdfium or opencv any time soon. We have to reckon with the fact that whatever we do, we will eventually get such a message.
The result of this is that the message is then retried, the server restarts, picks ups the message, crashes, and so on ad infinitum. Nothing gets processed till we manually stop the server, and purge the message. Not ideal.
What can we do to solve this problem?
What we don't want to do is create another service that monitors the rabbitmq service, looks for such messages and purges them, since that just leads to spiralling complexity. Instead we want to deal with this at the rabbitmq client level. We would be perfectly happy to say that if a message is not processed 3 times, we should just fail the message. We could do this by maintaining a database entry of which messages we've processed, but ideally I wouldn't want to involve anything external, and just contain the solution to this problem in our rabbitmq client library. I'm not sure how to do this though.
One method I have used in my event driven architecture is to use dead letter exchanges (DLXs) or poison queues, that way if we see the same message multiple times due to service failure then it'll be pushed into the DLX instead of being re-queued into the original exchange. These messages then trigger a different type of process within our system to alert us messages are stuck and failing to process, we can then diagnose and fix the consumer. After a fix has been made we trigger another process to move the poison messages back into the original exchange to be then processed as normal.
In your scenario because your process crashes there is two possible options to deal with these messages:
If the message is marked as redelivered then clone the message and add an attempt count to the body or as a header (x-attempt-count) to the message. The copy will then be added to the back of the queue with the attempt count. When the copy is then consumed you can check if it hits the threshold and then move the message into a DLX or store in a database. The major drawback here is that it breaks the order of which the messages are processed.
Use an external services to keep track of the number of delivery attempts, I would recommend using something like redis/memcache where you can increment a counter based on a unique message id. At the start of your process if the message has been marked as redelivered then lookup the counter. If the message has reached the threshold, trigger a different process again like moving it into a DLX.
I have some windows application .exe which are run on my Domain server, i have a problem that if the .exe is stooped how can i get the notification that the .exe has stopped.
is there any solution thru i can manipulate my code with operating system and get notification thru mail or any other resource
We have critical windows services that must always be running at work and the way we handled this was to write a monitioring program that listens to communication from the critical services and sends out an email if any of them stop "calling in". Here are a few details (although a bit simplified):
1) We use MSMQ messages to have the computers running the services talk to the monitor. Each service writes to a specific queue and the monitor is set to read from those queues. Note that MSMQ has pros and cons--if you choose to use this method, be sure to read up on it a bit.
2) The critical programs write messages to the queue detailing what it is they are doing and if they have had any errors.
3) If it has been more than 20 seconds (adjust accordingly for your situation) and the services haven't had anything to do, they simply write a message to the queue saying that they haven't had anything to do for the last 20 seconds.
4) The monitor reads these messages, keeps track of how long it has been since it has heard from each queue and if any of them haven't sent at least an "I've been idle for 20 seconds" message within 30 seconds (note that this should be longer than the other time period), it sends an alert to our emails saying which service has been idle for too long. Similarly, if any service has had any critical errors, the monitor may report those right away, too.
ALTERNATELY
There are off-the-shelf programs you can buy to do some or all of this for you, but this solution has worked well for us. If you are interested in 3rd party tools, you may consider looking at Splunk, Big Brother, and Tripwire. I don't have much/any experience with these tools, so I'm not sure if they will do what you want or not.
You can make use of the OnStop() event in the ServiceBase as given here. Related discussion post is here
So we have this somewhat unusual need in our product. We have numerous processes running on the local host and need to construct a means of communication between them. The difficulty is that ...
There is no 'server' or master process
Messages will be broadcast to all listening nodes
Nodes are all Windows processes, but may be C++ or C#
Nodes will be running in both 32-bit and 64-bit simultaneously
Any node can jump in/out of the conversation at any time
A process abnormally terminating should not adversely affect other nodes
A process responding slowly should also not adversely affect other nodes
A node does not need to be 'listening' to broadcast a message
A few more important details...
The 'messages' we need to send are trivial in nature. A name of the type of message and a single string argument would suffice.
The communications are not necessarily secure and do not need to provide any means of authentication or access control; however, we want to group communications by a Windows Log-on session. Perhaps of interest here is that a non-elevated process should be able to interact with an elevated process and vise-versa.
My first question: is there an existing open-source library?, or something that can be used to fulfill this with little effort. As of now I haven't been able to find anything :(
If a library doesn't exist for this then... What technologies would you use to solve this problem? Sockets, named-pipes, memory mapped files, event handles? It seems like connection based transports (sockets/pipes) would be a bad idea in a fully connected graph since n nodes requires n(n-1) number of connections. Using event handles and some form of shared storage seems the most plausible solution right now...
Updates
Does it have to be reliable and guaranteed? Yes, and no... Let's say that if I'm listening, and I'm responding in a reasonable time, then I should always get the message.
What are the typical message sizes? less than 100 bytes including the message identifier and argument(s). These are small.
What message rate are we talking about? Low throughput is acceptable, 10 per second would be a lot, average usage would be around 1 per minute.
What are the number of processes involved? I'd like it to handle between 0 and 50, with the average being between 5 and 10.
I don't know of anything that already exists, but you should be able to build something with a combination of:
Memory mapped files
Events
Mutex
Semaphore
This can be built in such a way that no "master" process is required, since all of those can be created as named objects that are then managed by the OS and not destroyed until the last client uses them. The basic idea is that the first process to start up creates the objects you need, and then all other processes connect to those. If the first process shuts down, the objects remain as long as at least one other process is maintaining a handle to them.
The memory mapped file is used to share memory among the processes. The mutex provides synchronization to prevent simultaneous updates. If you want to allow multiple readers or one writer, you can build something like a reader/writer lock using a couple of mutexes and a semaphore (see Is there a global named reader/writer lock?). And events are used to notify everybody when new messages are posted.
I've waved my hand over some significant technical detail. For example, knowing when to reset the event is kind of tough. You could instead have each app poll for updates.
But going this route will provide a connectionless way of sharing information. It doesn't require that a "server" process is always running.
For implementation, I would suggest implementing it in C++ and let the C# programs call it through P/Invoke. Or perhaps in C# and let the C++ apps call it through COM interop. That's assuming, of course, that your C++ apps are native rather than C++/CLI.
I've never tried this, but in theory it should work. As I mentioned in my comment, use a UDP port on the loopback device. Then all the processes can read and write from/to this socket. As you say, the messages are small, so should fit into each packet - may be you can look at something like google's protocol buffers to generate the structures, or simply mem copy the structure into the packet to send and at the other end, cast. Given it's all on the local host, you don't have any alignment, network order type issues to worry about. To support different types of messages, ensure a common header which can be checked for type so that you can be backward compatible.
2cents...
I think one more important consideration is performance, what message rate are we talking about and no. of processes?
Either way you are relying on a "master" that allows the communication needs, be it a custom service or a system provided(Pipes, Message Queue and such).
If you don't need to keep track and query for past messages, I do think you should consider a dead simple service that opens a named Pipe - allowing all other processes to either read or write to it as PipeClients. If I am not mistaken it checks on all items in your list.
What your looking for is Mailslots!
See CreateMailslot:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365147(v=vs.85).aspx
We are developing a Windows Forms application that will be installed on about 1,000 employee pcs. Users may run multiple instances of the application at the same time. The clients are all on a single intranet.
Changes in the application may cause database record changes, which in turn must be communicated to the other clients so their UIs are updated.
Our team has talked about two different approaches:
1. Multicast packets
The source client modifies the records and then sends out a multicast packet with a payload that something has changed. The other clients receive this and fetch the data specified. We need to account for the cases when the packet is not received, falling back onto actively retrieving the data.
My question at this point is how does a client know it didn't receive a packet? (don't know what you don't know) Which brings us to some sort of event log with timestamps in the database, and UI controls track the last time they were updated. They come into focus, check their timestamps, and update as needed.
Someone else said the UI elements would just reload every time they come into focus (think modes in outlook, bringing controls to the front of a stack workspace with CAB). And that the multicast is to update the clients that their current context has changed. If they miss it they work with stale data until they change modes and come back.
2. WCF and Callbacks
Clients register with WCF contracts for callbacks over a tcp binding. The primary technical concern with this is the server maintaining many open sockets. We have read up on how it isn't open in the traditional sense, it is put to sleep for a maximum of 90 seconds and then re-established at that point. We also read about the maximum number of open connections a Windows 2003 Server machine can handle, and how to modify that in the registry.
If we have 1,000 open socket connections to a server is this going to fall apart?
If anyone has faced this same situation and tried or evaluated the WCF approach we would love to hear about it.
I have not implemented a situation like this. However, I would think that one of the duplex bindings would not necessarily have a high overhead.
It all depends on how much information the server needs to send back to the clients. I understand you said the information will be used for them to update their UI. However it seems possible that they may not all need the same amount of information at the same time. For instance, if information about the Western region has changed, all 1000 clients may want to know that there is a change, and they may all want to update summary-level information about the Western region, but perhaps only 1/4 of them may need to see the details of the change.
If this is the case, then I'd recommend that the callback only provide information about what has changed, mostly at a summary level. Let those clients who are interested in the details of the change ask for the details. You might even go as far as to provide all the details for the top one or two levels of hierarchy, then for the rest, just include information saying "this changed at time". That way, depending on the level of hierarchy being viewed by a particular client, the client could then ask or not ask.
If necessary, you could batch updates together. If the clients only need to be updated once per second, then you could accumulate the changes for the last second and send them all at once.
You may also want to use some of the Peer to Peer bindings for some tasks. Perhaps the clients in a particular area of your business would like to know a little about what each other are working on - that sort of thing.