Beating Spam Filters (part one)

I recently setup email notifications on one of my web applications. It wasn’t long before I ran into a significant problem with spam filters, however. I had hoped some basic tactics would be enough to minimize the problem, but I ended up going much further, implementing Domain Keys and DKIM digital signing. There was a lot of confusing and conflicting documentation out there, but once I had everything figured out it didn’t take long to set up.

Depending on the constraints of your project, there are easier and more reliable ways to tackle this problem than the one I adopted. I’ll discuss the solution I implemented along with several alternatives. Since there is a lot of material here, I plan on breaking up this post into several smaller posts.

Basic Setup

There are a few basic things you can do to ensure the correct delivery of your email. If you don’t know what you’re doing, spam filters will assume you’re an indiscriminate spammer. Fortunately, there are some easy things you can do to greatly increase your chances of getting through. Some of these may be more obvious than others. Nevertheless, you’ll need to test your email delivery with each of these rudimentary tactics in place before being able to determine if more advanced tools are necessary.

Headers

When you send an email, you’re also sending a bunch of information about your server and domain in what is called the header of the email. Most email clients will hide this information to you, but it is critical to spam filters.

First, you should check to make sure your domain passes a reverse DNS check. The IP address of an email’s origin is also sent in its header. Most servers will check to make sure the domain sending the email (@site.example) is associated with the IP address sending that email. You can check for yourself by using an online reverse DNS lookup. Odds are you do not have a record setup yet.

There are some guides that explain how to set this up. You will also want to change the hostname of your server to match that of your domain. Slicehost has instructions for both of these things on there servers.

Another essential is to make sure your headers match, in particular your from and reply-to lines. In Rails, that might look like this:

def mail(email)
  ..
  @from   = "Site <noreply@site.example>"
  headers    "Reply-to" => "noreply@site.example"
  ..
end

Check your headers on delivered mail to ensure they are not being overwritten by your mailer. If you do not know how how to view email headers, try these instructions.

Spam Blacklists

You will also want to make sure your IP address is not on any blacklist. While you may think this improbable, if you do not have a firewall configured properly, someone may be bouncing spam off your server. That’s what happened to this fellow:

OUCH! My VPS got black-listed! Somehow the Spamhaus XBL list decided to black list the IP address of the server. Probably because I had port 25 opened in the firewall and somebody has taken advantage of my lack-of-linux-sysadmin-skills to start relaying spams. CRAP! Yahoo outrightly refused any SMTP connection from my server because it was marked as a spammer-wannabe .

Additionally, if you have not had your address for long, you will not know what the previous owner did with it. No tactics will overcome a blacklist, so be sure to look into it. Luckily, you can usually have yourself removed from these lists without much difficulty. Check your status at Spamhaus. They also have instructions on how to request removal if you find yourself on a blacklist.

Content and Formatting

If you’ve come across a lot of spam before, you will have picked up on a certain pattern in the way these emails are formatted. Often they have lots of colored text, spell things poorly or hide content. Spam filters have picked up on this pattern as well, so you will have to leave the bright jumbo lettering out of your emails.

Don’t forget that content matters too. If you are launching a social network for “natural male enhancement”, you may be out of luck with spam filters. Do not make your emails overly formal (”Dear Sir”) or commercial (”free trial”) either.

Be sure any email you send is well formed. Avoid using colored, hidden or image text whenever possible, along with nonstandard capitalization, spacing or spelling. Finally, use HTML markup minimally and where you do, make sure it is properly structured.

In upcoming posts, I will discussion the various mail server options I explored and how I went about digitally signing my emails.

| Popularity: 28%