The Problem
I have a web service that saves a record in the database and, sometimes send out notification emails to a group of users letting them know an event occurred.
I am getting infrequent time-out errors from the client. Since the data are committed to the database, I think my problem is that sending the SMTP server sending the emails is taking longer than the timeout on the client.
The Need
What I am need to do is to send the email either in the background or add it to some sort of queue for sending later and return.
Constraints
Our site runs .Net 2.0 and IIS 6
I do not have admin rights to the server, although I do have file-system access to our web site. While I may be able to convince our server admin to install a custom windows service for me, I would prefer to avoid this if possible.
Our web server is old and slow and is shared with several other web sites.
This problem occurred in an important online data entry system, where downtime and errors cause political issues.
Ideas
I've looked at several solutions, but need some direction as to which way would be best.
I could spawn a thread to send the emails, but I don't know if that
would work, since the web service code would fall out of scope upon
return.
I could add the task to some sort of queue, and periodically send queued emails.
increase the timeout on the client side and ignore the problem.
Under #2, I've looked at Jeff Atwood's use of the HttpRuntime.Cache to simulate a windows service but am very concerned with the warning
You need to really be careful on the length of the task running.
Every new Task is a new Worker Thread and there’s a limited number of
those – as it “borrows” a thread from the managed thread pool.
An unresponsive web page is worse than the error I'm trying to solve.
What direction should I go?
The Web Service Code
[WebMethod(CacheDuration = 0)]
public static string SaveRecord(comRecord record, IList<QIData> qiItems)
{
using (WebDatabase db = new WebDatabase())
{
db.SaveRecord(record, qiItems, UserId, ComId);
if (qiItems.Count>0)
{
/*Then somehow invoke or queue the routine
db.SendQINotice(record,UserId, (int)ComId));
*/
}
}
}
Interesting - there are several ways to do it, but since you are on a web site, I would add an entry to an e-mail queue in a database and have another task send out the e-mail. Then you have the freedom to do some better error handling on the e-mail send if you need to without slowing down the web site. For instance, you could add some "transient" error handling to such an application. If you are interested in this approach, I can add to my response the "transient" error handler that I am using to retry on an exception to overcome some temporary error conditions.
Related
I came across a situation in my work environment. where i have wcf service which receives messages from client and store in db. Now my problem is suppose server was down for 10 mins these 10 mins messages should be stored in client at some place and client should check for availability of server for every 1 min.Is there any procedure that i could follow or any help would be appreciated .Thank you
binding :netTCPBinding
MSMQ does exactly what your first sentence says - when you send an MSMQ message, if it can't get the remote queue then it stays with the client and the built-in MSMQ service retries in the background. That way your message, once sent, is "safe." It's going to reach its destination if at all possible. (If you have a massive message volume and messages need to be stored for a long time then storage capacity can be an issue, but that's very, very unlikely.)
Configure WCF to send/receive MSMQ messages
I'd only do this if it's necessary. It involves modifying both the service and the client, and the documentation isn't too friendly.
Here's the documentation for MsmqBinding. Steps 3 and 4 for configuring the WCF service are blank. That's not helpful! When I selected the .NET 4.0 documentation those details are filled in.
I looked at several tutorials, and if I was going to look at this I'd start with this one. I find that a lot of tutorials muddy concepts by explaining too many things at once and including unnecessary information about other parts of the writers' projects.
The client queues its messages locally
If you don't to make lots of modifications to your service to support MsmqBinding. You could just implement the queuing locally. If the WCF service is down, the client puts the message in a local MSMQ queue and then at intervals reads the messages back from that queue and tries sending to the WCF service again. (If the WCF service is still down, put the message back in the queue.)
I'd just send messages straight to the queue and have another process dequeue and send to WCF. That way the client itself just "fires and forgets" if that's okay.
That way you don't have to deal with the hassle of modifying your service, but you still get the benefit. If your message can't go to the WCF service then it goes someplace "safe" where it can even survive the client app terminating or the computer restarting.
Sending and receiving messages in a local queue is much easier to configure. Your client can check to see if the queues exist and create them if needed. This is much easier to work with and the code samples are much more complete and on-point.
We have this module where user can register and they need to confirm their e-mail address.
Currently I used .NET to send e-mail. So after adding the record I will now call my email method. But what I noticed is there are times that the e-mail functionality does receive a timeout error and since I catch all the exception this error is presented to the user.
What I wanted is to send the e-mail on the background. If there are any mail connection timeout, the e-mail method will just retry sending the e-mail for probably a minute or two.
And I'm thinking of using the SQL mail to achieve this. But I'm not sure if its the best solution.
You have a few options:
Use SQL Server Database Mail to perform the heavy lifting around the email queuing and sending. The application would still be responsible for constructing the message and sending it to the database. Relay through a separate SMTP server (installing SMTP services directly on a SQL machine is probably not a good idea).
Use async email methods from your web application, e.g. SmtpMail.SendAsync(). This will allow you to handle errors without holding up the user (or interacting with the user), but it's not durable. If your web process restarts, the operation is lost.
Use a Windows service. This will be a simple, separate application which simply monitors the database for new email tasks and/or is sent a message from the web application that a task needs to be performed.
Option #2 using SendAsync() will be the quickest to implement. Depending on your needs, that may be sufficient.
Option #1 shouldn't require much more effort. Option #3 really shines when you need to implement complex business logic around the recipient list and the email contents. It's also possible to combine options #1 and #3 (see the conversation between #RemusRusanu and me in the comments).
With both option #1 and #3, if the environment goes down, a service restarts, or an error occurs, you won't have lost any data or failed to send an email.
Rather manage the time-out in the .net application. If there is an issue or a time out in the trigger your data may not be committed. Rather allow the DB to complete the data transaction, and check the state from your .net App.
I dont think there is a definitive answer to this, but this would be my preference.
I have part of a project whereby I need to send 1 to up to 2000 emails from a webpage. One open source bug tracker used a Thread to send emails but it was riddled with bugs.
So just want to know if anyone has any ideas of how I can create a procedure that send up to 2000 emails a go?
I was told by one of my colleagues to leave it as one process (normal routine sending one email at a time) as it won't take long to process all the emails anyway. Doesn't seem right and I want to use threading... maybe I like complex routines?
[Extra]
I have something like:
foreach(string email in emailAddresses)
{
MailMessage mailMessage = new MailMessage();
mailMessage.To.Add(...)
...
SmtpClient client = new SmtpClient();
client.Send(mailMessage);
}
Use System.Net.Mail to send the messages; however you should use .NET 4 to avoid any connection issues, as there was a bug filed on the Connect website that will cause your messages to not get sent.
Don't use threading for three reasons:
Reason 1: A MTA is made to handle message retries and can handle failures. Your code may not be robust enough to handle this. System.Net.Mail is not able to do this out of the box.
Reason 2: If you do use threading, you will overwhelm the target SMTP server and it will block you. Most Windows SMTP relays have a default block of more than 15 (or 25?) concurrent connections.
If you're dealing with Exchange 2010, or 2007, then there is a throttling feature that gets activated if you send more than x messages per minute. This is a per MTA setting that will need to be adjusted to permit your situation.
Reason 3: The preferred way to do this is to have a dedicated IIS SMTP server (or Exchange...) that allows concurrent connections at high volume. Just use Sys.Net.Mail to hand the delivery task to the mail infrastructure. Not much is needed for a mail infrastructure. Just have a MTA that allows you to relay and it will "smart host" on your behalf out to the internet.
More questions on how to set up the MTA can be answered # serverfault.
However
You may want to use threading if your sending an email from an ASP.NET webpage... or will otherwise block the UI. Other than that, I don't see a need to run concurrent threads for the email generation task.
Lastly, if you're sending the same message to many recipients, you can either use a distribution list or append many target recipients to the same message.
You probably don't want to send 2000 emails on a thread that is servicing http requests. The user that submitted the request will be waiting for the server to respond until the emails send, and that's one less thread available for processing requests from other users. If you have many such requests it could drag down server performance.
Instead, I would post the request to a message queue, and have a seperate service process items from the queue and send emails.
Creating background threads in the asp.net app is another possibility, but then you end up in a situation where either:
You have to have your own task queue , separate from the one used by the normal thread pool.
Your email tasks are competing with, and potentially starving, the tasks for serving http requests.
That being said, there are deployment scenarios (a shared server, customer deployments) where the introduction of a second process is not desirable. If you don't have those constraints, however, I would go with the "separate process" because it's easier to scale your web site if it's focused on serving UI requests and leaves "fulfillment" tasks to a different service.
I think 2000 will take a long time (if it's a web page and the user is waiting for the page). It depends on your implementation, but you're doing this frequently you might want to have a "email thread" that's already been created. If there are no emails to send, then the thread can be suspended and not be consuming any resources. Then when the emails need to be sent you fill up an email queue with those emails and wake up the thread to start sending (if it is not already doing so).
If you're sending more than about 50 at a time, you need to contract out to a service that does this for you. Otherwise your messages will end up in people's spam folders.
I am building a web site, and the client wants a newsletter "system" on it.
How do I send this kinds of mass (>1000) emails?
I read somewhere that using sendasync method of the smtpclient does the trick.
But it constantly gives me an "Email faliure" exception. And, I don't know what to do with that right there...
So, basically my question is, is it ok to send the emails using the SEND method of the smtpclient, but each mail in it's own thread.
for eg.
NewsletterEmail newsletterEmail = new NewsletterEmail(emailAdress[i], mailSubject, mailBody);
Thread t = new Thread(new ThreadStart(newsletterEmail.MakeAndSendEmail));
t.IsBackground = true;
t.Start();
i think , you should rethink about your startegey for sending bulk emails
Creating > 1000 threads is not a good idea , it may even crash your server or it may make your server respond very slow.
Tell your client about Constant Contact. They are going to handle this much better than you ever could. It's also cheaper than your time.
In the event that fails you have a couple options.
If they already have an email server, leverage that to do the email broadcasting. In other words, relay the mail through that server.
If you can't do that, go download a free email server. I've been using hMailserver. Set it up and relay through it.
If you can't do that, write your own SMTP processing engine. Don't attempt to send the emails directly from ASP.Net. Queue them up in a database and write a windows service to handle the mail broadcasting.
Sending emails can sometimes take several seconds per email. This could completely hose your website while it's trying to handle sending 1000 emails.
A number of mail servers are configured with grey listing, meaning that they require you to send the same email twice in order to prove you aren't a spammer.
Next, getting the DNS appropriately set up can be a PITA. Which is why I suggest constant contact. I have one client that took nearly 5 years to finally get their DNS configured; and yes, I gave them explicit instructions once a year on what to do. Reverse DNS is critical.
Another thing is that some recipient servers have a limit on the number of threads they will accept from you at once. Most mail servers are built to take this into consideration. If you cross that boundary, then the recipient servers will consider you a spammer and take appropriate action.
Another problem area is in sending to a bad address, over and over again. AOL and others will consider you a spammer just for this one thing.
Point is, you really don't want to write this yourself.
Your best bet would be to have a separate process send the emails. Either have it run on a schedule to check for emails that need to be sent (maybe store the emails in a table?), or, if you don't like the scheduled process idea, then you can have a console application that is started by your website.
Something else to keep in mind is that if you are sending too many emails in a short period of time, it becomes very easy to get black-listed and then none of the emails from your domain will make it to any servers that have you black-listed.
I've taken two approaches:
1) Lazy, sloppy approach of configuring the asp.net process timeout long enough to complete via Send
2) Create a console app that is spawned by the web app.
I built a similar system a year or two ago. There's so many things that can go wrong when you send an email programmatically. For this reason, do yourself a favor and seperate the messages from the process of sending them. This way, you can "teach" your system that "badEmail#goodDomian.com" should be ignored or other similar situations.
You can store the message, subject, or whatever level of seperation of data you desire in the database along with a flag for meta-data like "SentOn", "FailedOn", etc. I sent my message one at a time to allow for individual errors to be stored and/or handled. I used SmtpMail.Send(), but whatever method you choose should work as long as you built something smart and recoverable.
Have a look at the MailChimp API: http://www.mailchimp.com/api/gettingstarted/
I'm having an issue sending large volumes of emails out from an ASP.Net application. I won't post the code, but instead explain what's going on. The code should send emails to 4000 recipients but seems to stall at 385/387.
The code creates the content for the email in a string.
It then selects a list of email address to send to.
Looping through the data via a datareader it picks out the email address and sends an email.
The email sending is done by a separate method which can handle failures and returns it's outcome.
As each record is sent I produce an XML node in an XML document to log each specific attempt to send.
The loop seems to end prematurely and the XML document is saved to disk.
Now I know the code works. I have run it locally using the same SMTP machine and it worked fine with 500 records. Granted there was less content, but I can't see how that would make any difference.
I don't think the page itself times out, but even if it did, I was sure .Net would continue processing the page, even if the user saw a page time out error.
Any suggestions appreciate because I'm pretty stumped.
You're sending lots of emails. During the span of a single request? IIS will kill a request if it takes longer than a certain (configurable) amount of time.
You need to use a separate process to do stuff like this. Whether that's a Timer you start from within global.asax, or a Thread which checks for a list of emails in a database/app_data directory, or a service you send a request to via WCF, or some combination of these.
The way I've handled this in the past is to queue the emails into a SQL Server table and then launch another thread to actually process/send the emails. Another aspx utility page can give me the status of the queue or restart the processing.
I also highly recommend that use an existing, legit, third-party mailing service for your SMTP server if you are sending mail out to the general public. Otherwise you run the risk of your ISP shutting off your mail access or (worse) your own server being blacklisted.
If the web server has a timeout setting, it will kill the page if it runs too long.
I recommend you check the value of HttpServerUtility.ScriptTimeout - if this is set then when a script has run for that length of time, it will be shut down.
Something you could do to help is go completely old-school - combine some Response.Writes with a few Response.Flush to send some data back to the client browser, and this tends to keep the script alive (certainly worked on an old ASP.NET 1.1 site we had).
Also, you need to take into account when this script is being run - the server may well also have been configured to perform an application reset (by default this is set to every 29 hours in IIS), if your server is set to something like 24 hours and this coincides with the time your script it run, you could be seeing that too - although the fact that the script's logging its response probably rules that out - unless your XML document is badly formed?
All that being said, I'd go with Will's answer of using a seperate process (not just a thread hosted by the site), or as Bryan said, go with a proper mailing service, which will help you with things like bounce backs, click tracking, reporting, open counts, etc, etc.