SPF in practice: mapping out all your senders

Module 4: Deliverability: SPF, DKIM and DMARC in practice · Reading time: 12 minutes

Learning objective: After this lesson you can track down every service that sends mail on behalf of your domain, fit them all into a single SPF record, count the lookups recursively up to the limit of ten, and consciously choose between ~all and -all.

Key concepts: sender inventory, envelope sender (Return-Path), a single SPF record, include, redirect, limit of 10 DNS lookups (RFC 7208 section 4.6.4), void lookups (maximum 2), recursive counting, permerror, flattening, SPF for subdomains, SPF on a domain that doesn't send mail, ~all and -all

A quick recap

An SPF record is a TXT record on your domain that starts with v=spf1 and lists which servers are allowed to send mail on behalf of your domain. The structure, the characters + - ~ ?, and the three most common mistakes (two records, +all, more than ten lookups) are covered in the DNS course, lesson 4.3. Read that first if SPF is new to you. This lesson deals with the tricky part: in practice, hardly anyone knows exactly who sends mail on behalf of their domain.

Step 1: the inventory

SPF only works if all senders are listed. Miss one, and that mail ends up in spam or gets rejected as soon as you tighten DMARC. So work through this list. For an average small business, you'll quickly find five or six:

Two good ways to find them:

1. Read the headers. Send yourself an email from each system (an invoice, a password reset, a test newsletter) and look at Return-Path and Authentication-Results in the headers (lesson 1.5). This shows you which domain and server it came through.
2. DMARC reports. Set DMARC to p=none with a report address (lesson 4.4). After a few weeks, the reports (lesson 4.5) will show you every server that sent mail on behalf of your domain, including ones you'd forgotten about. That's the most reliable inventory.

Not every service belongs in your SPF

This is where things often go wrong. SPF looks at the envelope sender (the Return-Path), not the From address. Many newsletter and sending services send with a Return-Path on their own domain, for example bounce-123@mail.mailer.example. The recipient then checks the SPF record of mailer.example, not yours. An include of that service in your record then does nothing, except cost you a lookup.

What that service actually needs to pass DMARC is a DKIM signature with your domain (lesson 4.3), or its own Return-Path subdomain of yours (for example bounce.example.com as a CNAME to the service). So check the service's documentation to see what it asks for. If it says "add include: to your SPF", do that. If it only says "create these CNAME records for DKIM and the bounce domain", then nothing needs to change in your main record. Module 7 covers this in more detail.

Step 2: one record, and count carefully

All senders go into one record. Suppose your inventory turns up: your hosting provider's mail server, Microsoft 365 for a few employees, your accounting software, and a helpdesk. Then the record might look like this:

example.com.  3600  IN  TXT  "v=spf1 a mx include:spf.hosting.example include:spf.protection.outlook.com include:_spf.boekhoud.example include:mail.helpdesk.example ~all"

How many lookups is that? Count every a, mx, include, exists, ptr and redirect, and then also count everything in the records behind each include:

At 11, the recipient returns a permerror: the entire SPF record then counts for nothing, for any sender. So mail from your own mail server gets judged more harshly because the helpdesk costs three lookups. And the tricky part is: your record can be 9 today and 11 tomorrow without you changing anything, simply because a service expands its own record.

Also: a maximum of two void lookups

RFC 7208 also mentions void lookups: lookups that return nothing (the name doesn't exist, or has no SPF or address record). More than two of these also leads to a permerror. This mostly happens with old includes from services you no longer use that have removed their record, or with an a: pointing to a server that no longer exists. So clean up.

Step 3: getting back under ten

If you're over ten, there are four solutions, from best to least good:

1. Remove what isn't needed. Services you no longer use. Services that send with their own Return-Path (see above) and therefore don't belong in your SPF. a if your website itself doesn't send mail. mx if your MX points to a service that's already listed as an include in your record.
2. Use a subdomain. Have a service send from nieuws.example.com or factuur.example.com. That subdomain gets its own SPF record with only that service, and doesn't count towards the ten of your main domain. Alignment in DMARC (lesson 4.4) also works for subdomains by default.
3. Use ip4: and ip6: for your own fixed servers. These don't cost a lookup. Only do this for addresses you manage yourself and that don't change.
4. Flattening (replacing all includes with the IP addresses currently behind them). This brings the count down to 0, but the record stops being accurate as soon as a service changes its addresses, and large services do this regularly. Only do this with a tool that keeps it updated automatically, and preferably not at all.

redirect or include?

redirect= says: use the SPF record of that other domain instead of this one. It costs one lookup, just like an include. It's handy if you have many domains that should all get exactly the same policy: v=spf1 redirect=_spf.example.com on each domain, with the actual record kept in one place. A record with redirect= must not have an all mechanism, because then the redirect is ignored. Gmail uses this itself: gmail.com has v=spf1 redirect=_spf.google.com.

Step 4: ~all or -all?

This course's recommended approach, in line with the DNS course:

• Start with ~all (softfail). Mail from a server not listed in your record is treated as suspicious, but usually not rejected outright. If you forget a sender, the damage is limited.
• Move to -all (fail) once you're certain all senders are listed, for example after reviewing a few weeks of DMARC reports and finding no unknown sources left.

With DMARC in place, the difference is smaller than often thought. DMARC decides based on "SPF or DKIM passed and aligned". A softfail counts for DMARC simply as "not passed". So the real protection comes from a strict DMARC policy, not from -all. Some recipients reject on -all even before the DMARC evaluation; that's a reason to only choose -all once your record is truly complete.

A domain that never sends mail

Do you have domains that never send mail (an old brand name, a typo domain)? Protect them against misuse with:

example.org.         IN  TXT  "v=spf1 -all"
_dmarc.example.org.  IN  TXT  "v=DMARC1; p=reject"
example.org.         IN  MX   0 .

Here, -all is correct right away: no one is allowed to send mail. The last line is a null MX (RFC 7505): "this domain does not accept mail". example.com itself is set up exactly this way.

In practice: what it looks like in a panel

With a WebOké hosting package, the default is v=spf1 a mx include:webawere.nl ~all. If you add a service, you adjust that record (in DirectAdmin: DNS Management, the pencil icon next to the SPF record, the Include Domain field; in My WebOké: the SPF helper on the DNS tab). Leave include:webawere.nl in place as long as your mail or website runs through those servers. With other providers, the default include is named differently, but the principle is the same: add to it, don't create a second record.

Summary

• Start by making an inventory of all senders; DMARC reports and headers help with this.
• A service with its own Return-Path doesn't belong in your SPF; it needs DKIM with your domain instead.
• Count recursively: every include, a, mx, exists, ptr and redirect, including within includes. Over ten, or more than two void lookups, means permerror.
• Get back under ten by cleaning up and using subdomains; only use flattening with automatic maintenance.
• Start with ~all, move to -all once you're certain. Domains that never send mail: -all, DMARC reject and null MX.

Quiz

Exercise

Exercise Z4.2: Having your domain's SPF counted recursively. Enter your own domain (or sidn.nl if you don't have one). First count for yourself how many DNS lookups the SPF record costs, including everything within the includes, and enter the number, along with the ending of the record (~all, -all, ?all or +all). The tool will count it live afterwards, including void lookups, and show you the tree.

Exercise O4.2: Your sending inventory. Using the table from this lesson, make a list of all systems that send mail on behalf of your domain. For each one, note: which server or service it goes through, whether it needs to be in your SPF (its own Return-Path or not), and whether it signs DKIM with your domain. Send a test email to yourself from at least two systems and note the Return-Path domain and DKIM domain (d=) from the headers.

Further reading

The steps in these articles apply to WebOké's panels; with other providers it works in a similar way.

• Free DNS course 4.3: SPF: who's allowed to send mail on behalf of your domain?
• DirectAdmin guide 5.7: Setting up and adding to SPF
• How does the SPF record work in DNS?
• Adding or changing an SPF record in DirectAdmin
• WebOké manual 5.5: DNS helpers: MX, SPF, DMARC, CAA and SRV
• RFC 7208 (SPF), section 4.6.4 (limits) and 6.1 (redirect); RFC 7505 (null MX)