I have a web application from which emails should be sent after specific actions. I have some alternatives for handling this I'm not sure which one is the best.
The first is that, when a user does an action the email is being sent directly from the ASP.NET application. But I think this is not a really reliable system because if the SMTP server is down or something else happens, the user just gets a feedback that his action cannot be completed.
As an alternative I was thinking about implementing a queuing system for what I have some ideas:
Push emails to send, into a database table, and a service application periodically checks for new messages, then sends them. On successful send it marks the email task completed.
Use MSMQ for queing. In this case the whole email could be passed as a message; or the other way is to store the message with attachments into a db table, and pass only the data which is required to query the db table and send the message. In this case I don't have to deal with size limits of MSMQ (because of attachments).
something else, like a local WCF service to notify the service
Which way you think is the best?
Use MSMQ is not good solution since has a limitation of 4 MB of each size. http://blogs.msdn.com/b/johnbreakwell/archive/2007/08/22/why-is-there-a-4mb-limit-on-msmq-messages.aspx
Worse case scenario, if MSMQ is failed like it process throw error or suddenly shutdown, it will loss many message. In my case, this solution is good when hardware and software is instaled in almost ideal
Use database and window service is better since it is a simple and doesn't need much effort.
I usually use a combination of database and file. The database contains table to save a header information and a flag that message has been action (either success or error or else) and files contains message (either html or plain) and attachment in original format.
When process is run to send, it is quicker to assemble a message from files rather than from querying blob/clob.
Since they are using file system on the application, you can add hardware like server or components or else to add availibility of the system easily.
Database can be added too, but it will cost you more license in databse software.
I add a test send email after send email in x times to make sure it is works well; this test email is send to my self or dummy inbox and an application to check the test email that is the same email that send and receive. If it is the same, sending pending email will continue again
Another way if you are using MS Exchange, you can use message queue by utilize its web service to queue send. This is an easy way but you need license.
You can see on MSDN library how to utilize MS Exchange web service.
You can use an email server like hmail. In order to push emails into a queue, you can push them to a mail server. To do that, you can write a windows form application that has a timer object that checks every row that has a Status 0(not sent) in email table. When the thread sends it to the mail server, it will be marked as 1(sent).
You can also classify your emails if you use DB. Different actions can send different emails. You can store this info in DB also so that your windows form application thread will now which email template to send.
Related
Problem
Currently, our website is setup so that when an action is taken that requires an email to be sent, our website will make a call the the SMTP server to attempt to send an email. The problem lies when the SMTP server goes down for whatever reason. We don't store any outgoing emails in any fashion so if the email to be sent fails, it's is lost forever (not really as it can easily be regenerated, but we don't have a mechanics to let us know it failed, except Azure Application Insights). While we also have the website send the devs an email when exceptions occur, for obvious reasons, we will not receive those emails.
Goal
Our goals is to stop having our website send an email directly to the email relay server. Instead, implement a solution that would send emails and have the ability to recover should a problem occur.
Stop the website from sending the emails
Ability to recover from transient or side issues/exceptions
Log as much activity as possible concerning the email (send attempts/fails/etc)
Ability to recover the activity logs from potential transient or side issues/exceptions
Ability to re-trigger a email to be sent if ever necessary (optional)
Solution
I read a 3-part article that sounds like it would solve this issue and I'm currently developing it.
I'm building a process using Microsoft.Azure.ServiceBus Topics and Subscriptions to manage sending emails from our website. I've gone through many samples and have been successfully able to SendAsync() a Message, ReceiveAsync() a Message, and CompleteAsync() or AbandonAsync() it appropriately.
Side-note: I'm now exploring how to work with the RetryPolicy to see if this will help me defer a retry to a bit of a longer period, though I'm not sure if I can/should use it for that.
While most of the process has been built so far, so I can understand the underlying infrastructure, I'm also still within the planning phase to make sure we plan appropriately.
We are currently trying to figure out the best or most appropriate workflow for this process. We figured two Topics would be needed: one for the emails to be sent EmailTopic, and one for the logs to record LogTopic.
The reason for the LogTopic is to handle any transient issues when attempting to save a log activity to the database. For example: I successfully retrieve an email to be sent. I then attempt the send the email and log this attempt. The email gets sent successfully. I then try to log this activity, but the database just went down and I'd would not be able to log this activity. The second Topic should alleviate that, but what happens if that goes down?
Here's our current workflow:
Website inserts data into a database that defines the email to be sent (currently, we'll have a field for the Body which will be the email contents itself, another table to hold the Email Templates which will contain the contents around the Body field, along with from, to, CC, BCC, and file attachments)
Website sends a small Message to the EmailTopic with the MessageId of the inserted record
A Stateless Service Fabric Service listens for messages
Receive the Message, get all details from the database for the record
Build the SMTPClient and attempt to send the email to the SMTP server
Send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken (attempt to send email)
If successful, CompleteAsync() the Message and send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken: "email sent"
If unsuccessful, AbandonAsync() the Message and send a Message to the LogTopic with the MessageId, current date, current DeliveryCount, and action taken: "email failed to send" (after 10 attempts message would automatically be placed in the DeadLetterQueue
In this workflow, the LogTopic will contain all the actions taken and will be stored in the database when the message(es) is(are) received. Obviously, if messages are abandoned for any reason and sent the the DeadLetterQueue, we will have a process to try to insert them at a later point.
Questions
We thought about just storing logs to the database within the workflow, but the question "what if the db goes down in the meantime?" (hence when Azure Central US went down last week) came up so we decided to use this 2nd Topic. Obviously, if the Service Bus is down, we can't send this message and I don't know how to recover from that, except log ETWs and check them some other way. Should I be attempting a DB save first, and if that fails, send a Message to the Topic?
Are there too many things going on in this service and should I split some operations around?
Is there a flaws or missing items in the workflow itself that we aren't taking into consideration?
Should we be using 1 Topic and add a Label to the message so we know it's a log vs email to be sent? Maybe using filters (not sure how to properly do this or if it's appropriate for this workflow yet)?
Are we asking too many questions in this 1 SO post and should split each question apart?
I think the work flow can be improved in such a way that the failures can be tracked and resolvable. I am providing my solution in achieving this.
Service Bus Topics supports multiple subscribers for single sender. This is achievable with the help of Subscriptions.
Instead of sending the message to two Topics, You can create two Subscriptions under a Topic. Refer here for filtering messages into Subscriptions using Rules.
You can create different Rules for the Subscriptions. Once the message is sent to a Topic, the custom properties of the message gets validated against the Rules of each Subscription. Based on the result of the validation, the message will get into the Subscription which has the required Rule condition.
Lets say the Subscriptions created are Email and Log. The Stateless Service Fabric Service should listen for the messages form both these Subscriptions.
Website should send the message to the Topic with the custom properties for Email Subscription.
Whenever a message is received by the Stateless Service Fabric Service, it should start a thread to send the mail.
Once the email is sent successfully, a Success message should be sent to the Topic with the custom properties for Log Subscription. In case of failure in sending mail, the message in Email Subscription should be dead lettered and a Failure message should be sent to the Topic with appropriate custom properties for Log Subscription.
The Stateless Service Fabric Service which listens also for the messages from Log Subscription will create a thread to write to the database as and when a message is arrived. In case of failure in writing into database, the message in Log Subscription should be dead lettered.
You can monitor the count of the dead letter messages of both the Subscriptions to ensure there is no failure. If the count is greater than 0, there should be manual intervention in Resubmitting the dead letter messages to the Topic. There are may tools available in market to monitor and resubmit the dead letter messages or you can also develop a custom application for doing that.
I guess this workflow should work as expected. Once this is installed, the only thing that need to be taken care is the dead letter messages in both the Subscriptions.
We have this module where user can register and they need to confirm their e-mail address.
Currently I used .NET to send e-mail. So after adding the record I will now call my email method. But what I noticed is there are times that the e-mail functionality does receive a timeout error and since I catch all the exception this error is presented to the user.
What I wanted is to send the e-mail on the background. If there are any mail connection timeout, the e-mail method will just retry sending the e-mail for probably a minute or two.
And I'm thinking of using the SQL mail to achieve this. But I'm not sure if its the best solution.
You have a few options:
Use SQL Server Database Mail to perform the heavy lifting around the email queuing and sending. The application would still be responsible for constructing the message and sending it to the database. Relay through a separate SMTP server (installing SMTP services directly on a SQL machine is probably not a good idea).
Use async email methods from your web application, e.g. SmtpMail.SendAsync(). This will allow you to handle errors without holding up the user (or interacting with the user), but it's not durable. If your web process restarts, the operation is lost.
Use a Windows service. This will be a simple, separate application which simply monitors the database for new email tasks and/or is sent a message from the web application that a task needs to be performed.
Option #2 using SendAsync() will be the quickest to implement. Depending on your needs, that may be sufficient.
Option #1 shouldn't require much more effort. Option #3 really shines when you need to implement complex business logic around the recipient list and the email contents. It's also possible to combine options #1 and #3 (see the conversation between #RemusRusanu and me in the comments).
With both option #1 and #3, if the environment goes down, a service restarts, or an error occurs, you won't have lost any data or failed to send an email.
Rather manage the time-out in the .net application. If there is an issue or a time out in the trigger your data may not be committed. Rather allow the DB to complete the data transaction, and check the state from your .net App.
I dont think there is a definitive answer to this, but this would be my preference.
I have many applications across an enterprise environment and they all use different methods of sending emails. Some send directly through an exchange server, some queue up locally in an SMTP queue and others call a a web service that then sends the email.
I'm trying to decide on the best way to get guaranteed delivery of emails. If our Exchange server goes down, then the applications that send to it directly can no longer send emails, also any emails sent during the down time never get anywhere. I would also like to implement a universal templating solution that all applications can share.
Are there any pre-built solutions to this problem, or do you have an insight on how to handle this issue?
We solved this by creating a web service that sends all our emails. This web service uses the
System.Net.Mail.SmtpDeliveryMethod.PickupDirectoryFromIis
setting, which essentially saves the files to a spot on the disk, tries to send them via the main SMTP server, and if the server is unavailable, they sit in the directory until it BECOMES available.
Guaranteed delivery, as long as the web service is up. Since we have redundancy checks in place, this is almost never an issue. If it is, we treat it as an error in code and handle it.
edit - added
I forgot to mention that XSS is a concern even in an email, so be sure to use something like the Microsoft.Security.AntiXss library, which contains functions like GetSafeHtmlFragment to strip out potentially dangerous scripts before outputting html to an email.
http://msdn.microsoft.com/en-us/security/aa973814.aspx
http://msdn.microsoft.com/en-us/library/aa973813.aspx
I have heard good feedback about Postmark. Maybe a service like that could be solution as it has several integration points.
http://postmarkapp.com
We have used HMailServer (on windows platform) which is freeware. Configured the max retries to too many & used external smtp to relay the emails. our applications ques up emails on the HMailServer and that server relays it further. with the max retries configured to be many if at all the main smtp servers are down we can assure that the email are delivered - but though there is no gurantee if there is huge downtime with main smtp relay servers.
I do this by queuing email to a SQL server database. Any ACID compliant database will work or you can use MongoDB with 'safe mode' inserts but if you really need guaranteed then use SQL server or MySQL. This way if your mail gets into the database it is 'guaranteed' not to be lost and your app doesn't have to think about it. You could use a web service or just make a shared assembly with a static public method in a class to drop your email in the db for you.
Include a column for status like 'new', 'delivered', 'recipient mailbox temporarily full', which you can represent with numeric values and keep a TimeToSend column, which starts out as the time when the email is queued in the database.
Then you have a mail app, that you can have run once a minute as a windows scheduled task. Make it as a console app. When it loads, it checks if an instance of it already running and if there is one, it exits. When running:
1. Attempts to deliver each mail to mail server. Query the database for all mail where the TimeToSend is older than now.
2. If mail is delivered to mail server, mark it logical deleted.
3. If any mail can't be delivered, advance the TimeToSend column for them to 10 minutes from now.
4. Delete records from table that are logically deleted. You can do this in the app or you can do it by having a sql job do it.
As mentioned earlier, you can utilize a web service that you can usually POST JSON to using an HTTP request. Here are a bunch of choices:
PostageApp (Ours!)
SendGrid
PostmarkApp
Amazon SES
They all have different feature sets and offerings, so definitely give them all a spin and figure out which you prefer.
(Full Disclosure: I am the Product Manager of PostageApp.)
I am building a web site, and the client wants a newsletter "system" on it.
How do I send this kinds of mass (>1000) emails?
I read somewhere that using sendasync method of the smtpclient does the trick.
But it constantly gives me an "Email faliure" exception. And, I don't know what to do with that right there...
So, basically my question is, is it ok to send the emails using the SEND method of the smtpclient, but each mail in it's own thread.
for eg.
NewsletterEmail newsletterEmail = new NewsletterEmail(emailAdress[i], mailSubject, mailBody);
Thread t = new Thread(new ThreadStart(newsletterEmail.MakeAndSendEmail));
t.IsBackground = true;
t.Start();
i think , you should rethink about your startegey for sending bulk emails
Creating > 1000 threads is not a good idea , it may even crash your server or it may make your server respond very slow.
Tell your client about Constant Contact. They are going to handle this much better than you ever could. It's also cheaper than your time.
In the event that fails you have a couple options.
If they already have an email server, leverage that to do the email broadcasting. In other words, relay the mail through that server.
If you can't do that, go download a free email server. I've been using hMailserver. Set it up and relay through it.
If you can't do that, write your own SMTP processing engine. Don't attempt to send the emails directly from ASP.Net. Queue them up in a database and write a windows service to handle the mail broadcasting.
Sending emails can sometimes take several seconds per email. This could completely hose your website while it's trying to handle sending 1000 emails.
A number of mail servers are configured with grey listing, meaning that they require you to send the same email twice in order to prove you aren't a spammer.
Next, getting the DNS appropriately set up can be a PITA. Which is why I suggest constant contact. I have one client that took nearly 5 years to finally get their DNS configured; and yes, I gave them explicit instructions once a year on what to do. Reverse DNS is critical.
Another thing is that some recipient servers have a limit on the number of threads they will accept from you at once. Most mail servers are built to take this into consideration. If you cross that boundary, then the recipient servers will consider you a spammer and take appropriate action.
Another problem area is in sending to a bad address, over and over again. AOL and others will consider you a spammer just for this one thing.
Point is, you really don't want to write this yourself.
Your best bet would be to have a separate process send the emails. Either have it run on a schedule to check for emails that need to be sent (maybe store the emails in a table?), or, if you don't like the scheduled process idea, then you can have a console application that is started by your website.
Something else to keep in mind is that if you are sending too many emails in a short period of time, it becomes very easy to get black-listed and then none of the emails from your domain will make it to any servers that have you black-listed.
I've taken two approaches:
1) Lazy, sloppy approach of configuring the asp.net process timeout long enough to complete via Send
2) Create a console app that is spawned by the web app.
I built a similar system a year or two ago. There's so many things that can go wrong when you send an email programmatically. For this reason, do yourself a favor and seperate the messages from the process of sending them. This way, you can "teach" your system that "badEmail#goodDomian.com" should be ignored or other similar situations.
You can store the message, subject, or whatever level of seperation of data you desire in the database along with a flag for meta-data like "SentOn", "FailedOn", etc. I sent my message one at a time to allow for individual errors to be stored and/or handled. I used SmtpMail.Send(), but whatever method you choose should work as long as you built something smart and recoverable.
Have a look at the MailChimp API: http://www.mailchimp.com/api/gettingstarted/
We are writing a feature to send a reminder email to customers in x number of days and just wondered if it was possible to delay the sending of the emails similar to how you can in Outlook (New Mail > Options button > Do not deliver before) in C#.
Does anyone know of a way of doing this?
If you want to submit the email to an SMTP server to hold for a number of days, the server would have to support this feature. I would recommend against doing it that way, however. If you had a problem with missing emails it could be hard to track down.
Here's what I would do:
Schedule the emails in your system by making a table entry.
Include a "SendDate" column in the table where you can set a date in the future
Write a Windows service that wakes up once every 15 minutes (for example) and sends the emails in that table where SendDate < currentDate
One possibility is to create a service that runs on a scheduled task processing 'pending' mail. You might store the pending mail in a SQL Database. (Dave details this)
This article looks promissing if you would like to avoid a 'real' service
http://www.codeproject.com/KB/aspnet/ASPNETService.aspx
No.
The way to accomplish this would be for your application to write the email to a queue (could be database, file, MSMQ, etc), and then write a separate process that, at a later date, reads mail from that queue and sends it.
As far as I know, the SMTP protocol doesn't have any kind of "delayed send" feature. Also, the feature you reference in Outlook actually just keeps it in your client's outbox until the designated date, so it's a client-side feature, not a server-side feature.
The SmtpClient is a pretty dumb client and doesn't support this -- sends are immediate whether they are synchronous or asynchronous. You may be able to do it using an Exchange integration. I'm not sure what APIs are exposed for Exchange, though. The simplest way would be to keep the message in a database table along with the time-to-send and have an off-line process that regularly scans this table and looks for messages whose time-to-send value has passed and are not marked as sent.