I've inherited an ASP.NET website written in c# using an mssql 2k8 database that is sending emails based on an insert into a message table via a trigger using database mail :| One of any failures and too many things rollback, aren't logged and no email is sent...
What a mess. I've written a few email subsystems in the past and I thought I'd ask for input before starting a rewrite here. What are the best practices for queuing/sending email in a Microsoft environment? Should I be pushing emails to a queue, from there pulling, sending, logging? DB Email seems like a fail. Is the queue managed in SQL server? Does SQL Server call a C# app? If an email send fails, what's a good approach for recovery?
Thanks for any insight!
I think you are correct to completely separate the use of system functions: use SQL for data, and push email on to a completely different 'service provider'; I assume you have some sort of business logic tier that will orchestrate this?
I can't talk in terms of 'best practice' from experience (for email specifically), but abstracting out the email service (just like you'd abstract out data-access) is definitely on the right path, and that's probably the critical decision you need to make right now. You could then have different implementations of the email (including SQL, if you really wanted to).
Depending on volumes - do you look at an asynchronous call or synchronous call? WCF seems like a good candidate to handle comms - this would allow you to fire data (for emails) into an end-point that had a queue built into it, or you could call (via WCF) a web service that acted synchronously.
You can send mail via sql-server. For more refer this
The architecture of it is here
Another implementation of sending mail via c# is this as they have developed an email factory for implementation... hope this helps
sp_send_dbmail places the mail request into a queue in msdb. After the transaction that sent the mail commits, the queue activates an external process that handles the SMTP delivery, including retries, logging and all that. Overall system is quite resilient, scalable and performant.
Perhaps you're using the old, deprecated xp_sendmail based system?
Related
I've been tasked with creating some sort of service that will take any e-mail sent to an e-mail address and handle the contents of the e-mail (including binary attachments.)
I've got full access to the server (Windows Server 2008) that will run this service, and have decided to code the solution using the .NET framework (3.5).
Now I'm thinking about the different ways I can do this. Since I'm not very experienced in handling e-mails programmatically, the first solution that came into my head was to create an e-mail client that will periodically poll an existing e-mail server for incoming e-mail (using the POP3 protocol) and process them. But there are other ways to do it.
One could be to use IMAP to stay connected to the mail server and fetch e-mails as soon as they arrive (more responsive, but I believe the IMAP protocol is harder to implement.) The follow-up question here is: where can I find reliable e-mail libraries that support the POP3 or the IMAP protocol?
Another could be to somehow configure the e-mail server to directly pipe the e-mails sent to a specific address to my binary code (I've seen this done on Linux servers.) I have no idea how to go about this, though.
The last I can think of would be to create a dummy e-mail server on its own address that handles the e-mail directly as it arrives, but to me this seems like a bad idea.
Does anyone have experience in this area? I would imagine that having to write e-mail handlers for ticket support systems isn't that uncommon, all I need in addition to that is to handle the attachments.
I'd be grateful for any hints and tips.
As with alot of things - it depends. Ask yourself the following questions:
What are your latency requirements--do you need to process incoming messages as quickly as possible, or can processing be batched? If it can be batched, then how often would you have to process the "inbox"?
What are your throughput requirements? How many messages are we talking about per minute here? How big are the messages? This would affect the decision you make about polling interval if using a batch scenario;
What sort of e-mail system are you integrating with? If it's Exchange, what programmatic interfaces are available to access a mailbox? Until the most recent version of Exchange, interestingly enough, there were issues with accessing a mailbox on an Exchange server (The client-side CDO COM components needed to be used which is not ideal---and there were security limitations).
By far the simplest approach is to poll a mailbox using POP3. However, if you need to respond immediately to an incoming message, then this isn't going to cut it.
As far as possible avoid writing your own SMTP service--it's been done a thousand times before and you're just creating unnecessary work for yourself and exposing yourself to security threats. If you absolutely have to respond immediately to messages, then rather set up an instance of Sendmail or Postfix to spawn a process that you have written.
If you're going to go for the POP3 solution (it looks like you are), then have a read of related questions "Free POP3 .NET library?" and "Reading Email using POP3 in C#".
I've used webdav in the past with c# to access an exchange server periodically and process emails.
This has worked quite well, and I'd probably use that method again if I need to do it.
Some of the .net components from http://www.quiksoft.com/ might help with your requirement.
The app polls a POP3 mail server every x minutes(s) and works it's way through the messages in the queue, and deletes them when it's processed each msg.
The QuikSoft tools also provide ways to parse the emails to get the content from each msg.
Receiving the email is not the hardest part, parsing, extracting attachments is.
If any one is interested in commercial product take a look at Mail.dll.
It supports IDLE command you have mentioned for instant notifications.
Mail.dll includes POP3, IMAP clients and powerful MIME parser:
using(Imap imap = new Imap())
{
imap.Connect("imap.server.com");
imap.Login("user", "password");
imap.SelectInbox();
List<long> uidList = imap.SearchFlag(Flag.Unseen);
foreach (long uid in uidList)
{
IMail mail = new MailBuilder()
.CreateFromEml(imap.GetMessageByUID(uid));
Console.WriteLine(mail.Subject);
}
imap.Close(true);
}
Please note that this is commercial product that I've created.
You can download it at http://www.lesnikowski.com/mail
We have this module where user can register and they need to confirm their e-mail address.
Currently I used .NET to send e-mail. So after adding the record I will now call my email method. But what I noticed is there are times that the e-mail functionality does receive a timeout error and since I catch all the exception this error is presented to the user.
What I wanted is to send the e-mail on the background. If there are any mail connection timeout, the e-mail method will just retry sending the e-mail for probably a minute or two.
And I'm thinking of using the SQL mail to achieve this. But I'm not sure if its the best solution.
You have a few options:
Use SQL Server Database Mail to perform the heavy lifting around the email queuing and sending. The application would still be responsible for constructing the message and sending it to the database. Relay through a separate SMTP server (installing SMTP services directly on a SQL machine is probably not a good idea).
Use async email methods from your web application, e.g. SmtpMail.SendAsync(). This will allow you to handle errors without holding up the user (or interacting with the user), but it's not durable. If your web process restarts, the operation is lost.
Use a Windows service. This will be a simple, separate application which simply monitors the database for new email tasks and/or is sent a message from the web application that a task needs to be performed.
Option #2 using SendAsync() will be the quickest to implement. Depending on your needs, that may be sufficient.
Option #1 shouldn't require much more effort. Option #3 really shines when you need to implement complex business logic around the recipient list and the email contents. It's also possible to combine options #1 and #3 (see the conversation between #RemusRusanu and me in the comments).
With both option #1 and #3, if the environment goes down, a service restarts, or an error occurs, you won't have lost any data or failed to send an email.
Rather manage the time-out in the .net application. If there is an issue or a time out in the trigger your data may not be committed. Rather allow the DB to complete the data transaction, and check the state from your .net App.
I dont think there is a definitive answer to this, but this would be my preference.
I have a client who would like a system developed which handles sending out alert emails to opted-in registered users. The alerts are based on geographic travel-related incidents and these are provided via a third party web service. The registered userbase is expected to be around 500,000 users and each user can subscribe to multiple alert categories.
I'm just trying to scope out what would be required technically to create this alerting functionality. In my mind we would create something like the following:
Poll the alert service once hourly
Store a "queue" of alerts in a temporary database table
Process the queue table as a scheduled task and send out the emails to the opted-in users
The part that I'm really not sure about is the physical sending of the emails. I was wondering if anyone could recommend any good options here. I guess we're looking to either create a custom component to send out the emails or use a dedicated third party service (like dotMailer maybe? Or a custom mass mail server?). I've read that rolling out your own solution runs the risk of having IP's blacklisted if you're not careful, and obviously we want to keep this all completely legitimate.
I should also mention that we will be looking to create the solution using .NET (C#) and we're limited to SQL Server Express 2008 edition.
Any thoughts much appreciated
Many thanks in advance
For the Poll , Queue and Send operations I'd create a windows service that calls the external service , operates on the data and then gathers the results to send out the emails updating the database as necessary with sent flags etc.
I handled a similar project recently and we found many ISPs or Hosting Providers got really twitchy when we mentioned mass emails. You should defintly check out the http://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003 CAN SPAM guidelines (or similar in your territory).
As long as you play by the rules and follow the guidelines you should be OK sending out from a local mail server however its important that you ensure that DNS Lookups or Reverse DNS lookups on the MX records all point back and forth to each other properly. Indeed this would be easier to out source to a third party mail provider or ISV but when we tried we were unable to find a good fit and ended up doing it ourselves.
Additionally you may want to glance at SPF records and other means to increase mass email delivery! For what its worth this can be a very tricky task to implement as SMTP (my least favourite protocol) is painful to try to debug and people get very upset if they receive multiples or unsolicited emails so ensure you have an Opt-in policy and appropriate checks to prevent duplicate delivery.
I have many applications across an enterprise environment and they all use different methods of sending emails. Some send directly through an exchange server, some queue up locally in an SMTP queue and others call a a web service that then sends the email.
I'm trying to decide on the best way to get guaranteed delivery of emails. If our Exchange server goes down, then the applications that send to it directly can no longer send emails, also any emails sent during the down time never get anywhere. I would also like to implement a universal templating solution that all applications can share.
Are there any pre-built solutions to this problem, or do you have an insight on how to handle this issue?
We solved this by creating a web service that sends all our emails. This web service uses the
System.Net.Mail.SmtpDeliveryMethod.PickupDirectoryFromIis
setting, which essentially saves the files to a spot on the disk, tries to send them via the main SMTP server, and if the server is unavailable, they sit in the directory until it BECOMES available.
Guaranteed delivery, as long as the web service is up. Since we have redundancy checks in place, this is almost never an issue. If it is, we treat it as an error in code and handle it.
edit - added
I forgot to mention that XSS is a concern even in an email, so be sure to use something like the Microsoft.Security.AntiXss library, which contains functions like GetSafeHtmlFragment to strip out potentially dangerous scripts before outputting html to an email.
http://msdn.microsoft.com/en-us/security/aa973814.aspx
http://msdn.microsoft.com/en-us/library/aa973813.aspx
I have heard good feedback about Postmark. Maybe a service like that could be solution as it has several integration points.
http://postmarkapp.com
We have used HMailServer (on windows platform) which is freeware. Configured the max retries to too many & used external smtp to relay the emails. our applications ques up emails on the HMailServer and that server relays it further. with the max retries configured to be many if at all the main smtp servers are down we can assure that the email are delivered - but though there is no gurantee if there is huge downtime with main smtp relay servers.
I do this by queuing email to a SQL server database. Any ACID compliant database will work or you can use MongoDB with 'safe mode' inserts but if you really need guaranteed then use SQL server or MySQL. This way if your mail gets into the database it is 'guaranteed' not to be lost and your app doesn't have to think about it. You could use a web service or just make a shared assembly with a static public method in a class to drop your email in the db for you.
Include a column for status like 'new', 'delivered', 'recipient mailbox temporarily full', which you can represent with numeric values and keep a TimeToSend column, which starts out as the time when the email is queued in the database.
Then you have a mail app, that you can have run once a minute as a windows scheduled task. Make it as a console app. When it loads, it checks if an instance of it already running and if there is one, it exits. When running:
1. Attempts to deliver each mail to mail server. Query the database for all mail where the TimeToSend is older than now.
2. If mail is delivered to mail server, mark it logical deleted.
3. If any mail can't be delivered, advance the TimeToSend column for them to 10 minutes from now.
4. Delete records from table that are logically deleted. You can do this in the app or you can do it by having a sql job do it.
As mentioned earlier, you can utilize a web service that you can usually POST JSON to using an HTTP request. Here are a bunch of choices:
PostageApp (Ours!)
SendGrid
PostmarkApp
Amazon SES
They all have different feature sets and offerings, so definitely give them all a spin and figure out which you prefer.
(Full Disclosure: I am the Product Manager of PostageApp.)
ASP.NET Webforms application.
Problem: How to implement emailling elegantly?
Many parts of the application demand it. This will include HTML emails, solutions how to handle bouncebacks etc..
I have the option to use my own SMTP server (app is hosted on a VPS), or to use the hosting providers.
I'd like to start logging here too (log4net probably)
Any suggestions for libraries to use / patterns to implement?
EDIT: http://martinnormark.com/2010/03/13/generate-html-e-mail-body-in-c-using-templates
I like having a debug catch asp.net net.mail - route all emails to a different email address in debug mode
For testing with a faked out SMTP server use ndumbster. Phil Haack has a blog post on usage - http://haacked.com/archive/2006/05/30/ATestingMailServerForUnitTestingEmailFunctionality.aspx
As you pointed out email templates are fantastic. Consider creating a template for HTML and plain text. Make sure your HTML email templates use basic HTML as a number of email clients out there are not up to modern day browser standards (eg. Outlook which uses the Word HTML rendering engine).
Log4Net is a good point to start at with logging. You can also use System.Diagnostics for simplistic tracing and debugging which can be written easily out to the event log or a file. I always prefer to have a wrapper for all logging so that logging frameworks can easily be swapped out if you find that you want to swap them out later because of a must have feature. It's also good from the point of testability as you can mock it out easily.
Really it doesn't matter which SMTP server you use as long as you don't get black listed. Make sure you check the SMTP servers IP against a DNSBL so you know if your SMTP host has a bad reputation. I highly recommend checking it using Barracuda Central's IP reputation - http://www.barracudacentral.org/lookups. Also make sure you're SMTP server does support resending when you're recipients use grey listing.
For bounce backs you can setup a POP account for it to send back to, it could be as simple as reading the emails and finding which email account the rejection came from and the subject of the previous email (which should be unique) so that you can send again later or after several bounce backs remove them from the list. To implement reading POP take a look at this thread - Reading Email using Pop3 in C#
The best pattern you can implement to save yourself loads of trouble down the road is not to send email directly from your web application. Instead, write the email (or the info you need to generate the email) to a database and have a separate process do the mailing.
Consider what happens when your smtp server is down and your app tries to send. If you have a separate process sending email you can send the email later when its back up.
Also you gain the ability to rate limit your outgoing email. If you end up with the Next Big Thing® on your hands you'll be glad you now have the ability to prioritize your outgoing mail or send mailing lists in batches instead of crushing your smtp server at the high point of your site's traffic.
If you are working with bouncebacks just have this process handle that as well. Now you have the ability to log a returned email address to a blacklist and check against the same list for every outgoing email. No way you want to have to put all that in your web app.
Regardless of what else you come up with please consider just writing the email to be sent to a database and letting your app go about its business. You can still get the message going out in near real time but you can get a real good separation of concerns with little effort.
As for incoming messages, I have had great success with the hMailServer which is a free e-mail server for Microsoft Windows. It supports the common e-mail protocols (IMAP, SMTP and POP3) and can easily be integrated with many existing web mail systems. It has flexible score-based spam protection and can attach to your virus scanner to scan all incoming and outgoing email. Definitely worth looking into.
For outgoing emails, I like the patterns I've expressed in this article: .NET MailMessage, LinkedResources, AlternateViews and Exceptions