We Keep Coding

How to handle mass email alert system for a registered userbase

I have a client who would like a system developed which handles sending out alert emails to opted-in registered users. The alerts are based on geographic travel-related incidents and these are provided via a third party web service. The registered userbase is expected to be around 500,000 users and each user can subscribe to multiple alert categories.
I'm just trying to scope out what would be required technically to create this alerting functionality. In my mind we would create something like the following:
Poll the alert service once hourly
Store a "queue" of alerts in a temporary database table
Process the queue table as a scheduled task and send out the emails to the opted-in users
The part that I'm really not sure about is the physical sending of the emails. I was wondering if anyone could recommend any good options here. I guess we're looking to either create a custom component to send out the emails or use a dedicated third party service (like dotMailer maybe? Or a custom mass mail server?). I've read that rolling out your own solution runs the risk of having IP's blacklisted if you're not careful, and obviously we want to keep this all completely legitimate.
I should also mention that we will be looking to create the solution using .NET (C#) and we're limited to SQL Server Express 2008 edition.
Any thoughts much appreciated
Many thanks in advance

For the Poll , Queue and Send operations I'd create a windows service that calls the external service , operates on the data and then gathers the results to send out the emails updating the database as necessary with sent flags etc.
I handled a similar project recently and we found many ISPs or Hosting Providers got really twitchy when we mentioned mass emails. You should defintly check out the http://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003 CAN SPAM guidelines (or similar in your territory).
As long as you play by the rules and follow the guidelines you should be OK sending out from a local mail server however its important that you ensure that DNS Lookups or Reverse DNS lookups on the MX records all point back and forth to each other properly. Indeed this would be easier to out source to a third party mail provider or ISV but when we tried we were unable to find a good fit and ended up doing it ourselves.
Additionally you may want to glance at SPF records and other means to increase mass email delivery! For what its worth this can be a very tricky task to implement as SMTP (my least favourite protocol) is painful to try to debug and people get very upset if they receive multiples or unsolicited emails so ensure you have an Opt-in policy and appropriate checks to prevent duplicate delivery.

Email sending strategy with C#/.NET

I have a web application from which emails should be sent after specific actions. I have some alternatives for handling this I'm not sure which one is the best.
The first is that, when a user does an action the email is being sent directly from the ASP.NET application. But I think this is not a really reliable system because if the SMTP server is down or something else happens, the user just gets a feedback that his action cannot be completed.
As an alternative I was thinking about implementing a queuing system for what I have some ideas:
Push emails to send, into a database table, and a service application periodically checks for new messages, then sends them. On successful send it marks the email task completed.
Use MSMQ for queing. In this case the whole email could be passed as a message; or the other way is to store the message with attachments into a db table, and pass only the data which is required to query the db table and send the message. In this case I don't have to deal with size limits of MSMQ (because of attachments).
something else, like a local WCF service to notify the service
Which way you think is the best?

Use MSMQ is not good solution since has a limitation of 4 MB of each size. http://blogs.msdn.com/b/johnbreakwell/archive/2007/08/22/why-is-there-a-4mb-limit-on-msmq-messages.aspx
Worse case scenario, if MSMQ is failed like it process throw error or suddenly shutdown, it will loss many message. In my case, this solution is good when hardware and software is instaled in almost ideal
Use database and window service is better since it is a simple and doesn't need much effort.
I usually use a combination of database and file. The database contains table to save a header information and a flag that message has been action (either success or error or else) and files contains message (either html or plain) and attachment in original format.
When process is run to send, it is quicker to assemble a message from files rather than from querying blob/clob.
Since they are using file system on the application, you can add hardware like server or components or else to add availibility of the system easily.
Database can be added too, but it will cost you more license in databse software.
I add a test send email after send email in x times to make sure it is works well; this test email is send to my self or dummy inbox and an application to check the test email that is the same email that send and receive. If it is the same, sending pending email will continue again
Another way if you are using MS Exchange, you can use message queue by utilize its web service to queue send. This is an easy way but you need license.
You can see on MSDN library how to utilize MS Exchange web service.

You can use an email server like hmail. In order to push emails into a queue, you can push them to a mail server. To do that, you can write a windows form application that has a timer object that checks every row that has a Status 0(not sent) in email table. When the thread sends it to the mail server, it will be marked as 1(sent).
You can also classify your emails if you use DB. Different actions can send different emails. You can store this info in DB also so that your windows form application thread will now which email template to send.

Email Anonymization Similar to Craigslist in C#

I am developing a site for which I would like to protect buyers by anonymizing their email addresses.Similar to craigslist's system, when a seller needs to contact a buyer they should be able to send an email to an anonymized address such as 1425415125#mysite.com which will then be routed to the user's email address.
My plan right now is to:
Set up a bucket (catch-all) inbox
Generate a random key for each buyer which will be the user specific ('1425415125' above) section of the email address
Monitor the bucket inbox and parse out this user specific section. Once I know the user, the email can be forwarded to the correct address
My questions are as follows:
Can you see any issues with the above solution
Are there any open source solutions to the existing problem
Are there any gotchas that one should be aware of when developing such a system?
Thanks in advance
JP

I did something related, though not quite the same. I setup a catch all inbox on my existing pop3 server (you probably have one already I'm guessing). I then used OpenPop.NET to read all new messages on a timer (say every 30 seconds). In my case I stopped at just processing the messagse but it's easy enough to generate a new message to the appropriate address and copy the body over, then send the new message out on your SMTP server.
One problem I see with your setup, and maybe it's just a misunderstanding on my part, is that while you are protecting the users original email address they will continue to be reachable at 1425415125#mysite.com basically forever. If I understand the way craigslist works, each posting has a different email address, and once the posting has been deleted/removed (or shortly after) the email address stops working. This makes it so that people can't just keep bugging you on that email address. The solution to this issue is easy, just make the email address coorespond to a post id or some other id rather then the users id in the database. The lookup will be just as quick but they will have a new email address each time.

You may wish to look at mail "piping" - the ability for someone to send an email to a mail server, which then gets thrown immediately to an executable, which then forwards your mail onto the recipient (by pulling the real email address from the database based on the incoming address from the piped message).
My personal recommendation would be to check out HMailServer, which has a COM API (the admin side is written in PHP, hence the requirement for legacy interop), is free and open-source, and is very well-documented. It doesn't have mail piping built-in, but is easily extensible given the API and support for scripts which run on server-side message events
HTH,
Benjamin

I think this solution will make sense and is in use in a lot of cases. The hardest part is actually receiving the messages. You can actually handle all of this within your web app if you need to. I wrote a blog post highlighting a couple of ways to receive email in your web app. It applies mainly to Rails but the concepts should be transferable.

The way you are lookimg to do it is the way I created a similar service. I would not recommend you writing your own smtp server. Use an existing mailserver and just use polling or some event based api.
The benefits of using a 3rd party mailserver is you can use existing backup and management tools on it.
Edit: I just noticed this has beed answered here with a better explanation. Pipe incoming email to a script on Windows IIS SMTP?

I do not see any problem with your setup, infact that is correct way to do because if your scheduled application fails, the emails will still be in the catch-all email box. Only once the email has been successfully delivered to somebody, the email should be deleted. You will be able to monitor and log the activity of your own application to monitor progress and failures.
I do not recommend Piping because, if for any reason piping goes successfully but your exe crashes, you will loose the email. Tracking will be difficult. Scheduling the jobs will not be possible.
If your application is independent of mail server, it is easy to manage it and replace your mail server whenever possible. It is easy to extend.
In this you will have to use some pop reader library, and schedule your application to run frequently.

In addition to email, you may consider a pull rather than push delivery mechanism, e.g: a message center web frontend or RSS feed. I say this because deliverability problems to various ISPs can be very difficult to troubleshoot and in my experience your users will never believe it's their ISP.

send email via asp.net 3.5

Im looking to send email vi a .net 3.5 form (C#)
Ive seen a few posts elsewhere and got the thing running ok for certain addresses but not for others, eg gmail accounts.
The finished app will have to pick up addresses from a database so Im having to cater for a lot of possibilites.
Im not looking to have my work done for me, just a shove in the right direction would be cool!
thanks
DD

Is it possible the gmail accounts are blocking you as a spammer? If you're sending out a lot of emails, or if there is a configuraiton issue on your server, they may have blacklisted you.
If the emails are going out from your server OK, but not reaching ALL of the recipients but is reaching SOME, then there is likely an issue outside of your direct control. An email goes through several servers in between your server and the enduser's inbox (and that's the case even if the inbox is on google's server for gmail.)
That said, there are things you can do to reduce your likelihood of being blocked/blacklisted.
This is a good place to start: http://www.andreas-kraus.net/blog/tips-for-avoiding-spam-filters-with-systemnetmail/
and then here is where to go from there. (Sorry - it's just a google search, but I gave you the relevant terms, at least.) http://www.google.com/search?q=system.net.mail+blacklist+spammer&rls=com.microsoft:en-us&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1
Finally, here's a very good resource with a cool URL. Perhaps the force will be with you if you learn these ways, young padawan.
http://www.emaildeliveryjedi.com/avoid-spam-filters.php
But getting back to your original question - everything you need to know about how to write code to send an email is covered here: http://www.systemnetmail.com/ It's the same for all versions of .NET from 2.0 on uo through the current 4.0.
And I'm sure you're already aware, but in case you're not, be sure you're familiar with the CAN-SPAM act. http://business.ftc.gov/documents/bus61-can-spam-act-compliance-guide-business

In addition to the great info from David Stratton...If you're certain that the gmail accounts are not receiving your emails, i.e. you've waited for sometime, checked the gmail spam boxes, and you've checked your SMTP server logs to ensure that Google is not sending any particular messages back to your SMTP server, then you may have run into a case of grey-listing / black-listing.
Black-listing can occur when a mail service has decided that the IP address of your mail server is a spamming server. It can happen under different scenarios, shared scenarios can be quite common.
One shared scenario: Let's say you recently acquired a new IP address, but it was recycled by your hosting provider from a prior client of theirs, who was a spammer. There is no guarantee that the IP address would be white-listed.
Another shared scenario: You're sending emails on a shared hosting website, where your SMTP service that you are using is shared with other clients of the SMTP service provider. It may be that one of those clients used the SMTP service maliciously or without following appropriate SPAM rules that gmail abides by.
Your best bet is to contact Google to commence a white-listing process. Providing you follow their requirements, they should eventually unblock emails originating from your service.

Delaying the sending of emails in C#

We are writing a feature to send a reminder email to customers in x number of days and just wondered if it was possible to delay the sending of the emails similar to how you can in Outlook (New Mail > Options button > Do not deliver before) in C#.
Does anyone know of a way of doing this?

If you want to submit the email to an SMTP server to hold for a number of days, the server would have to support this feature. I would recommend against doing it that way, however. If you had a problem with missing emails it could be hard to track down.
Here's what I would do:
Schedule the emails in your system by making a table entry.
Include a "SendDate" column in the table where you can set a date in the future
Write a Windows service that wakes up once every 15 minutes (for example) and sends the emails in that table where SendDate < currentDate

One possibility is to create a service that runs on a scheduled task processing 'pending' mail. You might store the pending mail in a SQL Database. (Dave details this)
This article looks promissing if you would like to avoid a 'real' service
http://www.codeproject.com/KB/aspnet/ASPNETService.aspx

No.
The way to accomplish this would be for your application to write the email to a queue (could be database, file, MSMQ, etc), and then write a separate process that, at a later date, reads mail from that queue and sends it.
As far as I know, the SMTP protocol doesn't have any kind of "delayed send" feature. Also, the feature you reference in Outlook actually just keeps it in your client's outbox until the designated date, so it's a client-side feature, not a server-side feature.

The SmtpClient is a pretty dumb client and doesn't support this -- sends are immediate whether they are synchronous or asynchronous. You may be able to do it using an Exchange integration. I'm not sure what APIs are exposed for Exchange, though. The simplest way would be to keep the message in a database table along with the time-to-send and have an off-line process that regularly scans this table and looks for messages whose time-to-send value has passed and are not marked as sent.