I'm writing a C# program that monitors a dedicated Gmail account using POP3 for specialized command emails and reacts appropriately.
For maximum reliability, I will run this program on several computers throughout the country.
I currently have a race condition where two instances of the program can read the same message before one of them deletes it, causing the message to be processed twice.
How can I make sure that each command is processed exactly once?
Gmail's POP3 access used to only serve each message once (making RETR and DELE a single atomic operation), but I can no longer reproduce this behavior.
The only method of communication between the computers is a SQL Server and an HTTP server (which I control).
One option I've thought of is to use POP3's UIDL command, and have a table in SQL Server with a unique column of UIDLs that were already processed.
Then, before downloading each message, the daemon would INSERT the UIDL into the table, and, if it got an error, skip the message. (I'm assuming that SQL Server's INSERT command is an atomic operation).
First I must admit I do not know what commands POP3 supports, but... if you can do an explicit 'DELE' and get an error if the message no longer exists, then I'd say:
RETR the message
DELE the message
Process only if DELE succeeded
EDIT:
After reading RFC1939, this approach should work; from the RFC:
DELE msg
Arguments:
a message-number (required) which may NOT refer to a
message marked as deleted
Restrictions:
may only be given in the TRANSACTION state
Discussion:
The POP3 server marks the message as deleted. Any future
reference to the message-number associated with the message
in a POP3 command generates an error. The POP3 server does
not actually delete the message until the POP3 session
enters the UPDATE state.
Possible Responses:
+OK message deleted
-ERR no such message
Examples:
C: DELE 1
S: +OK message 1 deleted
...
C: DELE 2
S: -ERR message 2 already deleted
This is ofcourse assuming that the Gmail implementation actually honours the RFC.
Related
What is the best way to be notified instant when mails or changes happens to the INBOX of my mailbox using MailKit??
I have been playing around with following events:
ImapClient.Inbox.CountChanged; This seems to work best when using Exchange servers, since they don't react at all on the MessageArrived event. Pff..
ImapClient.Inbox.MessagesArrived; This seems to work quiet well with open source mail servers, like SquirrelMail, but doesn't work at all with Exchange.
I want to be notified when new mails arrive to the mailbox, if any mails are moved/deleted, and if there any mails moved to this imap folder. Which approach should I take to get an event asap when something happens to my INBOX?? I want the best of both worlds.
And what is the ImapClient.Inbox.Subscribe(); used for??
The Subscribe and Unsubscribe methods just flag a folder (aka mailbox) as subscribed or unsubscribed - generally this is only used by mail clients to decide whether or not the user wants to see the folder in the default folder list.
The CountChanged event is emitted anytime MailKit gets an untagged "* # EXISTS" line from the IMAP server, typically as part of a response to a command that was just sent.
The MessagesArrived event is emitted immediately following the CountChanged event IF AND ONLY IF the new message count is larger than the old message count. Unfortunately, this is a badly designed/named event because it can be very misleading. Since the logic that determines whether or not to emit the event only has limited context (the old message count and the new message count), it can't accurately decide whether or not to emit this event.
Take the following situation for example:
When you open a folder, let's say that the message count is 10.
Now you expunge a handful of messages (5?) out of the folder... BUT, while the messages are being expunged, you get 2 new messages.
When the expunge command completes, the IMAP server replies back with "* 7 EXISTS"
Since 7 (the new message count) is less than 10 (the old message count), the MessagesArrived event will not be emitted.
I want to be notified when new mails arrive to the mailbox, if any mails are moved/deleted, and if there any mails moved to this imap folder.
If you want to know when new messages arrive, you need to listen to the CountChanged event and do your own book keeping to figure out if new messages arrived or if messages were moved/deleted out of the folder.
There's no way to distinguish between messages being moved into the folder vs new messages being delivered to the folder (unless you are doing the moving).
Which approach should I take to get an event asap when something happens to my INBOX??
If your server supports the IDLE extension, you'll probably want to look into using the ImapClient.Idle() (or IdleAsync()) method since the CountChanged event will only be emitted when it receives a "* # EXISTS" response from the server, and an IMAP server will only send that response as part of a response to a command from the client or if the client is in the IDLE mode.
If your server does not support the IDLE extension, you will need to "ping" the IMAP server using the ImapClient.NoOp() method (which is a dummy command that doesn't do anything) periodically in order to check if any new messages have arrived since the last command you sent.
There's an ImapIdle sample in MailKit's GitHub repository to see how to use it.
I am using WMQ to access an IBM WebSphere MQ on a mainframe - using c#.
We are considering spreading out our service on several machines, and we then need to make sure that two services on two different machines cannot read/get the same MQ message at the same time.
My code for getting messages is this:
var connectionProperties = new Hashtable();
const string transport = MQC.TRANSPORT_MQSERIES_CLIENT;
connectionProperties.Add(MQC.TRANSPORT_PROPERTY, transport);
connectionProperties.Add(MQC.HOST_NAME_PROPERTY, mqServerIP);
connectionProperties.Add(MQC.PORT_PROPERTY, mqServerPort);
connectionProperties.Add(MQC.CHANNEL_PROPERTY, mqChannelName);
_mqManager = new MQQueueManager(mqManagerName, connectionProperties);
var queue = _mqManager.AccessQueue(_queueName, MQC.MQOO_INPUT_SHARED + MQC.MQOO_FAIL_IF_QUIESCING);
var queueMessage = new MQMessage {Format = MQC.MQFMT_STRING};
var queueGetMessageOptions = new MQGetMessageOptions {Options = MQC.MQGMO_WAIT, WaitInterval = 2000};
queue.Get(queueMessage, queueGetMessageOptions);
queue.Close();
_mqManager.Commit();
return queueMessage.ReadString(queueMessage.MessageLength);
Is WebSphere MQ transactional by default, or is there something I need to change in my configuration to enable this?
Or - do I need to ask our mainframe guys to do some of their magic?
Thx
Unless you actively BROWSE the message (ie read it but leave it there with no locks), only one getter will ever be able to 'get' the message. Even without transactionality, MQ will still only deliver the message once... but once delivered its gone
MQ is not transactional 'by default' - you need to get with GMO_SYNCPOINT (MQ transactions) and commit at the connection (MQQueueManager level) if you want transactionality (or integrate with .net transactions is another option)
If you use syncpoint then one getter will get the message, the other will ignore it, but if you subsequently have an issue and rollback, then it is made available to any getter (as you would want). It is this scenario where you might see a message twice, but thats because you aborted the transaction and hence asked for it to be put back to how it was before the get.
I wish I'd found this sooner because the accepted answer is incomplete. MQ provides once and only once delivery of messages as described in the other answer and IBM's documentation. If you have many clients listening on the same queue, MQ will deliver only one copy of the message. This is uncontested.
That said, MQ, or any other async messaging for that matter, must deal with session handling and ambiguous outcomes. The affect of these factors is such that any async messaging application should be designed to gracefully handle dupe messages.
Consider an application putting a message onto a queue. If the PUT call receives a 2009 Connection Broken response, it is unclear whether the connection failed before or after the channel agent received and acted on the API call. The application, having no way to tell the difference, must put the message again to assure it is received. Doing the PUT under syncpoint can result in a 2009 on the COMMIT (or equivalent return code in messaging transports other than MQ) and the app doesn't know if the COMMIT was successful or if the PUT will eventually be rolled back. To be safe it must PUT the message again.
Now consider the partner application receiving the messages. A GET issued outside of syncpoint that reaches the channel agent will permanently remove the message from the queue, even if the channel agent cannot then deliver it. So use of transacted sessions ensures that the message is not lost. But suppose that the message has been received and processed and the COMMIT returns a 2009 Connection Broken. The app has no way to know whether the message was removed during the COMMIT or will be rolled back and delivered again. At the very least the app can avoid losing messages by using transacted sessions to retrieve them, but can not guarantee to never receive a dupe.
This is of course endemic to all async messaging, not just MQ, which is why the JMS specification directly address it. The situation is addressed in all versions but in the JMS 1.1 spec look in section 4.4.13 Duplicate Production of Messages which states:
If a failure occurs between the time a client commits its work on a
Session and the commit method returns, the client cannot determine if
the transaction was committed or rolled back. The same ambiguity
exists when a failure occurs between the non-transactional send of a
PERSISTENT message and the return from the sending method.
It is up to a JMS application to deal with this ambiguity. In some
cases, this may cause a client to produce functionally duplicate
messages.
A message that is redelivered due to session recovery is not
considered a duplicate message.
If it is critical that the application receive one and only one copy of the message, use 2-Phase transactions. The transaction manager and XA protocol will provide very strong (but still not absolute) assurance that only one copy of the message will be processed by the application.
The behavior of the messaging transport in delivering one and only one copy of a given message is a measure of the reliability of the transport. By contrast, the behavior of an application which relies on receipt of one and only one copy of the message is a measure of the reliability of the application.
Any duplicate messages received from an IBM MQ transport are almost certainly going to be due to the application's failure to use XA to account for the ambiguous outcomes inherent in async messaging and not a defect in MQ. Please keep this in mind when the Production version of the application chokes on its first duplicate message.
On a related note, if Disaster Recovery is involved, the app must also gracefully recover from lost messages, or else find a way to violate the laws of relativity.
I Am writing Pop3 client for gmail. The problem is that when the email is downloaded then it deletes from pop3 server. I have tried sending RSET command to server but it dose not work.
Do i have to send RSET just before disconnecting or after downloading each mail?
It should only delete the messages if you send/use the DELE command. Since you're writing it yourself.. simply do not use the DELE command: http://www.faqs.org/rfcs/rfc1939.html
The RSET command should also work, too, see from the RFC linked:
RSET
Arguments: none
Restrictions:
may only be given in the TRANSACTION state
Discussion:
If any messages have been marked as deleted by the POP3
server, they are unmarked. The POP3 server then replies
with a positive response.
Possible Responses:
+OK
Examples:
C: RSET
S: +OK maildrop has 2 messages (320 octets)
Gmail's POP3 configuration maybe sometimes confusing.
In the Gmail's web interface in "Settings" on "Forwarding and POP/IMAP" tab you can find a drop down list called "When messages are accessed with POP".
Here's the value list and the Gmail's behavior:
a.
"delete Gmail's copy":
Message is deleted by issuing RETR command.
b.
"keep Gmail's copy in the Inbox":
Message stays in the Inbox,
Web interface says it's unread, however message is not received
for the second time using POP3 client.
c.
"archive Gmail's copy":
Message is deleted by issuing RETR command, however it is
possible to find it using web interface.
Gmail simply ignores DELE command, and does not follow POP3 standard.
Google doesn't handle POP3 in a conventional sense. Once you have downloaded a mail then it is flagged as downloaded and POP3 will act like it has been deleted. But in reality they keep a copy of the mail on the server.
It has been some time, but working on a similar problem, i found out that the TOP command does not mark the message as read within GMail POP3 server.
So, if you do not read the message entirely, you can surely read it back later.
Hope it helps.
Because Gmail POP3 server doesn't work like normal POP3 server, it hides old emails automatically even the email was not deleted by POP3 DELE command.
If you want to read all emails from Gmail mailbox, you can try to implement or find a IMAP4 client. I can use IMAP4 to read all emails in Gmail mailbox.
I was wondering if anyone could help me out (not with code, although that would be appreciated), with the logic behind checking and retrieving messages from a POP3 mail server.
I.e.
Establish connection
Validate credentials
Enumerate message list
Check each message to see if it's "new"
Download "new" message(s).
Would this be the correct way about doing this?
Thank you
The best way of looking at something like this is to have a look what something else does. Run Wireshark or some other packet capture software, and use an e-mail client to check. Anyway, the basics of a POP3 session are as follows:
USER username
PASS password
LIST <-- Shows the size of each waiting message
UIDL <-- Shows a unique ID for each waiting message
RETR 1 <-- Retrieves message with index 1
DELE 1 <-- Deletes the message you just retrieved
QUIT
The first char of all of the responses except RETR will be a + (success) or a - (failure).
If you are deleting messages off the server after retrieving them, you don't need to bother with UIDL. If you are leaving them, you can use UIDL to get a unique ID for each message which you store locally to show you have retrieved that message before.
For more details, see the RFC. Wikipedia also lists a more in depth example, showing the server response.
These should be useful:
POP3 Email Client (.NET 2.0)
POP3 Client as a C# Class
Retrieve Mail From a POP3 Server Using C#
POP3 Sequence Diagram
POP3 Reference
I am using System.Net.Mail to to send mail. I do not know the type/version of the SMTP relay that it will connect to.
Some errors will result in no email being sent (e.g. no addresses or invalid from address) whilst others errors will still result in an email being sent.
E.g. Send To : bob#somewhere.com CC: fred#somewhere.com and DoesNotExist#somewhere.com may result in an error of
The server response was: 550 #5.1.0 Address rejected DoesNotExist#somewhere.com.
But the email still appears to be delivered to Bob and Fred.
Is there any reference to which error codes will still result in an email being delivered or any programmatic way of determining this?
This is the way that SMTP operates, as described in the RFC.
Failure to deliver to one recipient does not affect delivery to other recipients.
Generically, codes starting with 4 and 5 are failures, codes starting with 2 are success codes - see RFC821 section "4.2.2. NUMERIC ORDER LIST OF REPLY CODES".
"Email being delivered" is hard to define. Email being accepted for delivery by the SMTP server is defined by the SMTP protocol (and the server will signal acceptance to deliver a message) but this server might just relay the message to another server, or operate in a smart host configuration where it just accepts messages and passes them on to the smart host - delivery is another thing and it's usually associated with local delivery (LMTP).
Some mail clients work around this problem of not being able to tell if a message has been delivered by implementing the dreaded read receipts - but this implementation is entirely on the client side and AFAIK it has nothing to to with SMTP.
RFC 821 is the one that describes SMTP and includes information about different types of errors, but not sure if it contains the details you're looking for. And either way, even if you can find out that the mail server accepted the email for some users, that is not the same as saying that it was delivered to them.
As far as I know, there is no way of telling if an email has been delivered except if the recipients mail client will tell you in some way.
Non-delivery reports (NDRs) are system messages that report the delivery status of a message to the sender. The messages are a subclass of a general message information structure that is referred to as delivery status notifications. Delivery status notifications describe three different types of situations:
* Success (2.X.X numeric codes)
* Persistent transient failure (4.X.X numeric codes)
* Permanent failures (5.X.X numeric codes)
To learn more about delivery status notifications, see Request for Comment (RFC) 1891 and RFC 1893.
Qouted from Microsoft Support http://support.microsoft.com/kb/284204