Encrypted delivery: TLS, MTA-STS and DANE

Module 5: Reputation and the major providers · Reading time: 10 minutes

Learning objective: After this lesson, you can explain how mail servers encrypt traffic between each other, why this isn't enforceable by default, and what MTA-STS, TLS-RPT and DANE change about that. You can look up and read the records.

Key terms: STARTTLS between servers, opportunistic encryption, downgrade attack, MTA-STS (RFC 8461), _mta-sts record, policy file, mode testing/enforce/none, TLS-RPT (RFC 8460), _smtp._tls record, DANE for SMTP (RFC 7672), TLSA record, DNSSEC

Two kinds of connections

Lesson 1.4 covered the connection between your mail program and your provider's mail server: port 587 with STARTTLS, or 465 with direct TLS, with a login. There, you're in control. Your program knows which server to expect, checks the certificate, and refuses to connect if something's wrong.

This lesson covers the next step: the connection between the sending server and the receiving server, on port 25. That works differently.

Opportunistic: encrypt when you can

When a server delivers mail, it connects to the recipient's MX. After EHLO, it checks whether the receiving server offers STARTTLS. If so, it switches to an encrypted connection. If not, it simply delivers the mail unencrypted.

This is called opportunistic encryption: encrypt when you can, otherwise don't. And usually the sending server doesn't strictly check the certificate. An expired or self-signed certificate isn't a reason to stop.

Why is it done this way? Because mail always has to keep working, even with old servers. A sender has no way of knowing whether a recipient is supposed to have TLS. That opens the door to two attacks:

1. Downgrade. Someone who can tamper with traffic along the way strips the word STARTTLS out of the recipient's response. The sender thinks there's no encryption and delivers unencrypted. The attacker reads along.
2. Fake MX. Someone who can forge DNS responses presents their own server as the MX. Because the certificate isn't checked, the sender notices nothing.

These days, almost all major mail servers encrypt, and Gmail requires senders to use TLS (lesson 5.3). But "usually encrypted" is different from "always, and with the right server". Two solutions fix that: MTA-STS and DANE. Both are set up by the recipient: they let others know that mail to your domain must always be encrypted, to a server with a valid certificate.

MTA-STS: policy via a web page

MTA-STS (Mail Transfer Agent Strict Transport Security, RFC 8461) works with two parts.

A TXT record that says a policy exists:

_mta-sts.example.com.  3600  IN  TXT  "v=STSv1; id=20261108T120000;"

And a policy file at a fixed location, over https with a valid certificate:

https://mta-sts.example.com/.well-known/mta-sts.txt

version: STSv1
mode: enforce
mx: mx1.example.com
mx: mx2.example.com
max_age: 604800

The clever part: a sender remembers the policy for the duration of max_age. An attacker who later strips the STARTTLS word or presents a fake MX gets caught out: the policy states that encryption is mandatory and which servers are valid. On the very first connection (before the policy is known), MTA-STS doesn't protect you; that's its well-known weak point.

A real example is gmail.com: the TXT record _mta-sts.gmail.com exists, and the policy file at https://mta-sts.gmail.com/.well-known/mta-sts.txt is set to mode: enforce.

TLS-RPT: reports on failed encryption

With MTA-STS set to enforce, mail can get stuck if something's wrong with your certificate. You want to know about that. That's what TLS-RPT (RFC 8460) is for:

_smtp._tls.example.com.  3600  IN  TXT  "v=TLSRPTv1; rua=mailto:tlsrapport@example.com"

Senders that support TLS-RPT (including Google and Microsoft) then send a daily report in JSON format: how many connections succeeded and which failed, with the reason (for example an expired certificate or a name not listed in the policy). Just as with DMARC reports (lesson 4.5), you can have these processed by a reporting service.

Order of rollout: set up TLS-RPT first, then MTA-STS in testing, read the reports for a few weeks, and only then switch to enforce. This mirrors the DMARC advice: observe first, then enforce.

DANE: the certificate in the DNS, secured with DNSSEC

DANE (DNS-based Authentication of Named Entities) takes a different approach. You add a TLSA record in the DNS for each MX server, with a fingerprint of the certificate or the key:

_25._tcp.mx1.example.com.  3600  IN  TLSA  3 1 1 0d5a5c8f3d8aab82bba32bf8d711fa2433e4fe5fa4828b0443364d3ab5cdc1d5

_25._tcp means: for SMTP on port 25. The three numbers say how to read the value. For mail servers, 3 1 1 is the most common: "this is the SHA-256 fingerprint of the public key of this exact server". 2 1 1 also occurs: that holds the fingerprint of an intermediate certificate.

A sender that supports DANE sees the TLSA record, requires encryption, and checks whether the server's certificate matches. If it doesn't match, it won't deliver.

Why is DANE safe against forged DNS responses? Because it only works with DNSSEC (DNS course, lesson 8.1). Without a valid DNSSEC chain, TLSA records are ignored. That's also the threshold: both your domain (for the MX) and the mail server's zone must be signed.

DANE doesn't have a "first time" weakness like MTA-STS, and it's widely used in the Netherlands and Germany. internet.nl tests for it. Microsoft supports DANE for incoming mail in Exchange Online. The MX servers of internet.nl also have TLSA records.

Pitfall: if you replace a mail server's certificate with DANE using a new key, the TLSA record must change along with it, and in advance: add the new record first, wait until the TTL has expired, and only then start using the new key. Otherwise, senders using DANE will refuse your mail. With 3 1 1, this only matters if the key changes, not with every renewal that keeps the same key.

MTA-STS or DANE?

You can use both at the same time. RFC 8461 says that a sender supporting DANE gives DANE priority if valid TLSA records exist.

Who arranges this?

You set this up for incoming mail on your domain. If your mail runs through a hosting company or mail service, it depends on that party:

• DANE is only possible if the provider publishes TLSA records for its mail servers and your domain has DNSSEC. At WebOké, for example, you can enable DANE in the same place as DNSSEC (see the internet.nl article below); with other providers it works similarly or is arranged by default.
• MTA-STS you can sometimes set up yourself, but then mta-sts.example.com needs to be a website with a valid certificate, and you need to know your provider's MX names.
• TLS-RPT you can always set up yourself: it's just a TXT record with a report address.

For your outgoing mail, you don't need to do anything: your provider's server decides whether it respects MTA-STS and DANE from recipients.

Summary

• Between mail servers, TLS is opportunistic by default: encrypted when possible, without strict certificate checking. That opens the door to downgrade attacks and fake MX records.
• MTA-STS: TXT _mta-sts plus a policy file at https://mta-sts.<domain>/.well-known/mta-sts.txt; mode testing or enforce.
• TLS-RPT: TXT _smtp._tls with a report address, for daily reports on failed encryption.
• DANE: TLSA record at _25._tcp.<mx-server>, only works with DNSSEC and also protects the first connection.
• Roll it out like DMARC: reports first (TLS-RPT, testing), then enforce.

Quiz

Exercise

Exercise Z5.4: Looking up MTA-STS, TLS-RPT and DANE. Look up the MTA-STS record and the TLS-RPT record for gmail.com, open the policy file and fill in the mode (enforce, testing or none). Then look up the TLSA record for an MX server of internet.nl at _25._tcp.<mx-server> and fill in the first three numbers of one of the records. Optional: the same questions for your own domain (there, the answer may be "none"). The tool looks everything up live and compares.

Further reading

The steps in these articles apply to WebOké's control panels; with other providers it works similarly.

• Internet.nl checklist
• What is DNSSEC?
• Free DNS course 8.1: DNSSEC in plain language
• internet.nl, test your email (independent test of STARTTLS, DANE and more): https://internet.nl/
• RFC 8461 (MTA-STS), RFC 8460 (TLS-RPT), RFC 7672 (DANE for SMTP)