Return to sender – Florian’s beginner guide to improve e-mail deliverability (I)

How you can improve e-mail deliverability by working on your IP and domain reputation

If you run your own mail server, you’ll sooner or later likely run into e-mail deliverability problems due to what is called IP and domain reputation. Especially larger providers have various and sometimes not easy to understand filtering systems in place that might refuse messages from your machine.

While there is no magic wand to solve all deliverability problems, this article will give you an overview of some essentials that help messages from your mail server hopefully reach their destination.

Although I’ve been running my own mail servers for the past two decades now, I’m neither a mail service provider, nor do I host mailboxes on a large scale. The problems and findings I’m dealing with are likely very much different from what large e-mail service providers have to face, and there are clearly people out there who are much more expert on this topic than me.

I want to share what I’ve learned, to give help and guidance for postmasters who operate small servers, to help them avoid running into the same issues and repeat the same mistakes I experienced. That being said, I’m more than happy for any feedback, advice, insight, correction and also criticism - if you have anything to share that I can incorporate into this article, don’t hesitate to reach out!

This article will be split in two parts. This first part explains the general topic and gives some advice and best practice, while in the second part I will introduce you to the concept of DNS-based blackhole list (DNSBL), feedback loops (FBLs), DNS-based whitelists (DNSWL), and share a set of services to check your IP address and domain against, to help remedy delivery issues.

Better be safe than sorry

You absolutely should ensure your machine is configured properly. This article does not cover the installation of a mail server per se - please refer to the excellent guides from Christoph Haas or Thomas Leister instead. It’s also advisable to get a deeper understanding on how a mail server works by reading one of the many available books on the subject.

Please be aware that e-mail is very critical for most businesses these days, and managing a mail server can be a very challenging task. Mistakes can lead to severe problems that can even impact your business operations. If being a postmaster is not amongst your core competences, and you’re not up for learning how things work, do consider putting this into the trusted hands of someone who knows how to do the job. There are many e-mail providers out there, so choose wisely - I prefer to go with someone local instead of the big players, to work with someone for whom privacy and safety are key concerns, and ideally someone who shares their knowledge with the community via books, lectures or articles.

What by now you should have in place already, and what this article doesn’t cover, is:

an IP address that is not in a regular dial-up space, as these will be blocked by most mail servers straight away
setup of corresponding A, AAAA and PTR records
implementation of DKIM and SPF
setup of postmaster@ and abuse@ aliases
a proper server configuration that prevents you being an open relay
spam and optionally antivirus filtering, also for outbound e-mails
sender login maps, to prevent sending e-mails with fake accounts
ideally, also a DMARC record and a DNSSEC-secured zone
remove invalid recipient addresses from newsletters and mailing lists as soon as they bounce
optionally, also rate limiting for both incoming and outgoing e-mails

What’s this reputation thing about?

The so called reputation of your IP address can be bad, even if you didn’t send out a single e-mail yet. How can that be?

First, inexpensive virtual servers, as they are offered nowadays, are an invitation to spammers to cheaply rent machines, drop off their spam, and then move to the next provider to repeat that. This lowers the reputation of whole provider’s networks

if you are in a “spammy” neighbourhood, also your address can be negatively impacted. Think of this a bit like the credit scoring that banks do - if you’re in the noble villa district your ranking will likely be better than in the cheaper areas of town.

Second, as we run out of IPv4 addresses, it is unlikely that your address has never been in use before - and the Internet hardly forgets. If the previous owner of that IP address used it to send spam, even months or years later and despite new ownership, the reputation can be bad. Think of this as taking over a restaurant or hotel with a bad reputation - even if you change the name, people might not know the owner changed, as the address is the same, and will avoid going there, unless you advertise a real change happened.

Third, especially large providers with thousands or millions of mailboxes seem to move to a very careful approach, and distinguish between known IP addresses and fresh ones, and measure their usual mail rate. If the known pattern changes, the filters might get triggered. Some providers even require the unblocking of IP addresses that haven’t been used for outgoing messages in a certain amount of time.

Long story short: Even if you didn’t do anything wrong, you might run into problems to deliver your e-mail to certain destinations. If you are actually sending out spam yourself, the situation gets even worse - hacked accounts, infected computers or insecure webmailer forms can quickly degrade your IP’s reputation and have a negative impact on your delivery rate.

Last but not least, if you are a source of backscatter, or forward messages to external accounts that are spam, your IP reputation can likewise be negatively impacted.

Is the IP address really that important?

The reputation of the sending IP address is one of the main parameters for spam filtering. However, at least recently, several providers also seem to factor in a domain-based reputation. In other words, if your domain name has a good reputation, it could even be reflected after a change of IP address - similar to a restaurant that is widely recognized and moves to a different street. On the other hand, this means that a bad reputation will not change just because you switch your IP address.

Some senders of mass newsletters seem to be using a dedicated domain (and separate IP address) possibly for this purpose, to not have their newsletter reputation negatively impact their main address.

Fear of loss

Especially in today’s cloud-based hosting, it’s trivial to switch from one server to another. Need more space? Upgrade the machine. Want a different dataserver? Move the machine. Want a dedicated CPU instead of a virtualized one? Bump up your server plan.

However, this comfort has its price - often, a change of machine means also a change of IP address, so all the precious work you have done to build reptuation is gone and you need to start from zero, which can be an unrewarding and unpleasant experience.

Luckily, serveral providers noawadays provide a solution for that and offer you to book a dedicated IP address or a full subnet that you can flexibly attach to your machines, as long as you stay with the same hoster. This feature e.g. is called “floating IP” or “failover IP” and usually costs a few additional Euro per month, but is definitely worth it.

Just connect that second IP address to your server, configure the mail server hostname on it, and tell the mail server to use it for outgoing connections.

Does a bad reputation always result in blocked messages?

There are at least five different ways a negative IP reputation can affect the deliverability of your e-mails:

deferred e-mail, temporary rejection (soft bounce): Similar as with e-mail greylisting, your messages will be temporarily rejected. Some larger e-mail providers implement this mechanism also when a new IP address delivers messages to their servers, or when the amount of messages sent varies too much from previous patterns. After your IP address gained a good reputation over time, restrictions are often lifted. E-mails affected from a temporary rejection can be delayed by a few hours, but eventually should reach the recipient. Depending on your server’s configuration, senders get informed about delayed e-mails after a certain amount of time, but usually not immediately. The SMTP error code for this temporary rejection starts with 4xx.
rejected e-mail, permanent rejection (hard bounce): Your e-mails are immediately rejected and will not be resent. Senders get immediately informed about that. The SMTP error code for this permanent rejection starts with 4xx.
add to spam spam score, add subject prefix, add header: Not all messages get delayed or rejected. Instead, the recipient’s server can accept them, but raise the message’s spam score, together with other measures, like adding a “[SPAM]” prefix to the subject or adding a note directly to the message header or body itself. That per se does not affect deliverability, but some e-mail clients might locally flag these messages as spam then.
move to spam folder: Even if your e-mail reaches the destination server, there’s no guarantee yet it will show up in an user’s inbox. The antispam filters can also decide to accept the e-mail, but put it into the spam/junk folder. In general, I strongly discourage the use of this kind of filtering - it suggests the sender that the e-mail has been accepted, while the recipient might just not look into the spam folder and therefore never reads the e-mail.
silent deletion: I never experienced this myself, but hearsay suggest that some providers at least did this in the past - accepting the e-mail, but instead of delivering it into the inbox or at least the spam folder, just silently delete the message without notifying neither sender nor recpient. I strongly recommend not to do so - it might even have legal implications if e-mails just “vanish” that way. Think of this as a letter taken from the postbox by your colleague, but directly trashed by them, so you never see it.

So, how do I learn of delivery issues?

Unfortunately, there’s no easy way to learn of all potential delivery issues, and it can take some time until you actually hit the problem. You’ll obviously spot the issue when you or your users are affected and get a “bounce”, but it’s always better to proactively identify issues and remedy them before they can actually occur.

To find out if your server already has issues to deliver messages, the pflogsumm tool comes in handy. The following simple command, that can also be run via cron each day, will show you delivery problems and bounces:

pflogsumm -d yesterday /var/log/mail.log --rej-add-from --verbose-msg-detail --problems-first --ignore-case --iso-date-time

That still only shows problems after the fact, but it’s a start. Another time-consuming, but helpful way to test for delivery issues is to actually create accounts for yourself at widely-used mail services, e.g. (in alphabetical order) GMail, GMX, Office 365, Outlook/Live/Hotmail, T-Online and Yahoo to send yourself an e-mail, e.g. via the swaks tool. The following command will send a standardized test message directly from the server:

swaks --to youraccount@mail.provider

Should this result in issue #4 from above (move to spam folder), you can try to mark it explicitly as “not spam” in the provider’s mail interface, send it again and see if the status changes. You can try to repeat this three or four times, so possible the mail gets directly delivered in your inbox. However, whether this spam filter training then applies only to your personal account, or will positively effect your reputation at large, depends on the provider. Be aware that this mechanism can also work the other way round - if too many e-mail recipients flag your messages as spam, your reputation can suffer.

Aren’t there any easier workarounds?

Sometimes I read the advice that it’s not worth the hassle to deal with larger mail providers, and instead route e-mails to these specific destinations through one of the commercially available SMTP relay services. I don’t have any hands-on experiences myself, so can’t comment if it actually brings a benefit. Should you decide to go that route, keep in mind privacy and GDPR requirements when certain mails are now routed through third party servers, and be aware that you need to adjust DKIM and SPF records for your domain to include the additional outbound servers.

To be continued…

In the second part of this article I will introduce you to the concept of DNS-based blackhole list (DNSBL), feedback loops (FBLs), DNS-based whitelists (DNSWL), and share a set of services to check your IP address and domains against, to help remedy delivery issues.