[ Transport security ]

TLS negotiation failures and unsupported protocol settings

Use TLS reports to locate negotiation failures, then reproduce the handshake safely before changing supported protocol settings.

By the EmailAuth research team · 4 min read

[ Key points ]

  • A generic failure does not identify a protocol-version problem.
  • A common TLS version is necessary but not sufficient for a handshake.
  • Retest the same SMTP path against an approved security baseline.
On this page

Use TLS reports to locate negotiation failures, then reproduce the handshake safely before changing supported protocol settings.

Do not infer a cipher problem from a generic label

References: RFC 8460 §4.3.1.

TLS-RPT has starttls-not-supported and a general validation-failure category, but a generic failure is not proof that the receiver disabled a particular protocol or cipher. RFC 8460 recommends failure-reason-code for additional context. Preserve that text with the reporting MTA and endpoint. Distinguish failure to advertise STARTTLS, failure to begin the handshake, certificate validation failure and failure after authentication. These stages require different evidence and should not all become a dashboard label such as TLS broken.

Verify the SMTP path rather than a web service

References: RFC 3207 §4.

SMTP STARTTLS upgrades an SMTP connection after protocol negotiation; testing an HTTPS endpoint does not exercise that path. Use an authorised diagnostic connection to the actual receiving host and service. Record the greeting, EHLO capability response, STARTTLS response and handshake result without sending real message content unnecessarily. Compare the implicated address and receiving route with production configuration. Middleboxes and alternate listeners can change what the sender sees, so a test from one network is evidence for that path, not every reporter’s path.

Use a compatibility model with honest limits

References: RFC 8446 §4.1.3, RFC 8446 §4.2.1.

The synthetic fixture assigns sender-supported versions {TLS1.2, TLS1.3} and receiver-supported versions {TLS1.3}. Their intersection is nonempty, so the simplified model has a common version. Change the sender set to {TLS1.2}; the intersection is empty. This demonstrates one necessary compatibility condition, not a successful handshake. Cipher suites, signature algorithms, certificate properties, groups and implementation policy can still prevent negotiation even when a version overlaps. Do not publish the model’s output as tested interoperability between named mail products.

Change settings from an approved security baseline

References: RFC 9325 §3.1.1.

Compare the actual negotiated failure with vendor documentation and your security policy. Avoid enabling obsolete protocols or weak algorithms solely because a report contains failures. If a legitimate sender cannot meet the receiving baseline, investigate upgrades and the operational impact with its owner. Changes to transport security require a deliberate risk decision, staged rollout and rollback plan, rather than an automatic reaction to an aggregate count. Keep the previous configuration and capture the exact setting that changed.

Validate repair without overclaiming coverage

References: RFC 8460 §4.3.4.

Repeat the same authorised path test after the change and record the negotiated version and relevant validation outcome. Exercise backup listeners and address families that participate in the affected route. Compare later TLS-RPT periods for the same reporter and policy domain, recognising that transient TCP and busy-network errors are not required to be reported by RFC 8460. An empty failure list therefore cannot prove the whole network is reliable. No production SMTP connection or production security-setting change was performed. The controlled loopback evidence below extends the set-intersection model with actual TLS handshakes.

Retained historical evidence and current checks

The transport observations below were captured on 7 November 2026 and inspected, not rerun as network tests, for this review. The new offline model checks are recorded separately.

The retained controlled SMTP STARTTLS capture tests a server requiring TLS1.3 against a client limited to TLS1.2. The handshake fails with TLSV1_ALERT_PROTOCOL_VERSION at the client and UNSUPPORTED_PROTOCOL at the server. A compatible positive control negotiates TLSv1.3. Recorded SMTP exchanges show that STARTTLS was advertised and accepted before the TLS failure. This distinguishes a protocol incompatibility from missing STARTTLS without changing production settings. It does not establish named-provider interoperability. The historical transport runner requires Python with cryptography and loopback sockets. Its outcomes are bounded lab evidence, not current service health or production interoperability.

Primary-source basis

The field definitions and reporting limits come from RFC 8460, SMTP TLS Reporting, sections 3 and 4. Where STS validation is discussed, consult RFC 8461. The operational recommendations are distinguished from protocol requirements.

[ Takeaway ]

Retest the same SMTP path against an approved security baseline.

[ By role ]

What this means for you

Choose your role for practical next steps.

IT / Mail Admin

Locate the handshake stage

Reproduce the named condition before making a configuration change.

Next steps

  1. Locate the failure within the SMTP STARTTLS exchange.
  2. Record actual negotiated settings on retest.
  3. Test alternate receiving routes implicated by reports.

CISO / SecOps

Protect the security baseline

Keep security conclusions proportional to authenticated evidence.

Next steps

  1. Do not enable legacy protocols automatically.
  2. Review cipher and signature constraints separately.
  3. Keep network visibility limits in the incident conclusion.

Founder / Non-technical

Balance compatibility and risk

Make the result useful to the person responsible for service continuity.

Next steps

  1. Make compatibility exceptions an explicit risk decision.
  2. Prioritise legitimate sender upgrade paths.
  3. Require measured impact before changing baselines.

[ FAQs ]

Frequently asked questions

Does validation-failure identify a TLS version?

No. It is a general category requiring more evidence.

RFC 8460 §4.3.1

What does starttls-not-supported indicate?

The reporter found that the recipient MX did not support STARTTLS.

RFC 8460 §4.3.1

Can HTTPS testing validate SMTP STARTTLS?

No. They are different protocol paths.

RFC 3207 §4

Does a shared version prove a handshake succeeds?

No. Other cryptographic and implementation constraints still apply.

RFC 8446 §4.1.3

Should old protocols be enabled automatically?

No. Use an approved baseline and investigate compatibility deliberately.

RFC 9325 §3.1.1

Are all network timeouts reported?

No. RFC 8460 does not require reporting transient network errors.

RFC 8460 §4.3.4

What must a retest preserve?

The endpoint, path, timestamp, settings and actual negotiation outcome.

Was a live MTA tested?

No production MTA was tested. A retained 7 November controlled SMTP STARTTLS capture documents actual TLS handshakes; this review reran only the offline compatibility model.

[ Sources ]

Sources

  1. RFC 3207

    Sections §4

  2. RFC 8446

    Sections §4.1.3 · §4.2.1

  3. RFC 8460

    Sections §4.3.1 · §4.3.4

  4. RFC 8461

    Sections §4.2

  5. RFC 9325

    Sections §3.1.1

[ Glossary ]

Glossary

STARTTLS
The SMTP command that requests an upgrade of the connection to TLS.

RFC 3207 §4

Version overlap
At least one TLS version offered and accepted by both peers; it does not prove all handshake parameters are compatible.

RFC 8446 §4.1.3 RFC 8446 §4.2.1

Vantage point
The network origin and path from which a diagnostic observation is made.

Enterprise provenance

Why top financial institutions and governments retired their email security dashboards.

Traditional email authentication platforms sell you a dashboard and ask your engineers to do the labour. In high-stakes environments, manual oversight means delayed enforcement, configuration drift and unmonitored shadow domains.

EmailAuth is the agentic email provenance platform. Our agents enforce p=reject, keep third-party senders aligned and move your domains to full RFC 9989 compliance, the new DMARC standard,, with human-in-the-loop auditability.

Governments Large enterprises Tier-1 financial institutions

SaaS · Dedicated tenancy · Air-gapped on-prem · BYOK encryption

Knowledge Base