[ Transport security ]

Certificate-name failures in SMTP TLS reports

Investigate certificate-name failures against the reported SMTP endpoint and applied policy, keeping policy-host HTTPS identity separate.

By the EmailAuth research team · 4 min read

[ Key points ]

  • Separate an allowed MX pattern from the certificate identity check.
  • For MTA-STS SMTP, require a matching SAN DNS-ID, not Common Name-only success.
  • Investigate reported paths and historical policy before broadening permissions.
On this page

Investigate certificate-name failures against the reported SMTP endpoint and applied policy, keeping policy-host HTTPS identity separate.

Identify which certificate was evaluated

References: RFC 8460 §4.3.1, RFC 8461 §3.3.

certificate-host-mismatch is a transport negotiation failure category in RFC 8460. Begin with policy-type, policy-domain, receiving-mx-hostname, receiving-ip and the reporting period. The report may describe an SMTP service, while MTA-STS policy retrieval uses a separate HTTPS hostname. A valid certificate on mta-sts.example.com does not prove that mx1.example.net presents a suitable SMTP identity. Conversely, fixing the policy website certificate will not repair an SMTP endpoint that presents the wrong certificate.

Check policy matching and certificate identity separately

References: RFC 8461 §4.1, RFC 8461 §4.2, RFC 8461 §3.3.

For MTA-STS, the selected MX host must match the policy’s permitted MX patterns and the SMTP TLS certificate must satisfy the required identity and trust checks. An MX pattern in the policy is not a certificate issued to that host. RFC 8461 §4.2 requires a matching DNS-ID in the SMTP certificate subject alternative name. As the OpenSSL hostname-check documentation explains, a generic hostname checker may accept Common Name fallback; that alone does not meet this SAN requirement. Check the actual MX target, not the recipient’s visible email domain or the server’s reverse-DNS label. Preserve the reported policy string because a sender can still be using an unexpired cached version. Compare historical and current configurations before deciding which change caused the mismatch.

Walk through the wrong-endpoint example

In a synthetic configuration, example.com has MX mx1.example.net and an STS policy permitting mx1.example.net. The SMTP endpoint presents a certificate whose only DNS SAN is web.example.net. The MX-policy comparison passes, but the exact hostname is absent from this example certificate identity, so the illustrated identity check fails. Replacing the SAN set in the local fixture with mx1.example.net makes that deliberately simplified exact-name check pass. The fixture is not a full X.509 validator and does not implement wildcard, chain, expiry or DANE rules.

Inspect every route that can present a certificate

Load balancers, alternate IP addresses and backup MX hosts can produce intermittent failures if only some listeners have the intended certificate. Collect the actual presented chain and endpoint identity through authorised SMTP STARTTLS tests. Compare the route and timestamp with deployment records. Do not disable hostname validation to call the test successful. Avoid testing only an HTTPS port or one preferred address and then generalising to all inbound mail paths. Check both address families where deployed and relevant to the affected reporter.

Repair the responsible layer and recheck

References: RFC 7672 §3.1.1, RFC 8460 §4.1.

Install the correct certificate or repair routing to the intended TLS listener. If an MX-policy pattern is genuinely wrong, review that change separately and consider sender caches. Do not add broad wildcard permissions merely to silence an unexplained failure. Retain before-and-after evidence, then compare subsequent reports by occurrence period rather than arrival time. A DANE policy requires its own usage-specific authentication analysis: DANE-EE identity is not the same name-check model as MTA-STS. This article’s worked example is explicitly STS and uses no production domain or live certificate.

Retained historical evidence and current checks

The transport observations below were captured on 7 November 2026 and inspected, not rerun as network tests, for this review. The new offline model checks are recorded separately.

The retained evidence/tls_lab.py runner and dated result document a real local SMTP STARTTLS handshake. The positive control trusts an ephemeral lab CA and validates mx1.example.test. The name_mismatch case uses a certificate for web.example.test and fails with certificate verify code 62. The timestamped result includes the SMTP exchange, exact validation message and certificate SHA-256 fingerprint; public_lab_certificates.json retains the public leaf for inspection. No name-check bypass is used. This retained capture documents a controlled endpoint failure but not any production receiving route. The historical transport runner requires Python with cryptography and loopback sockets. Its outcomes are bounded lab evidence, not current service health or production interoperability.

Primary-source basis

The field definitions and reporting limits come from RFC 8460, SMTP TLS Reporting, sections 3 and 4. Where STS validation is discussed, consult RFC 8461. The operational recommendations are distinguished from protocol requirements.

[ Takeaway ]

Investigate reported paths and historical policy before broadening permissions.

[ By role ]

What this means for you

Choose your role for practical next steps.

IT / Mail Admin

Compare identity and routing

Reproduce the named condition before making a configuration change.

Next steps

  1. Compare the reported MX with certificate SANs.
  2. Inspect all implicated listeners and addresses.
  3. Retain both reported and current STS policies.

CISO / SecOps

Avoid weakening validation

Keep security conclusions proportional to authenticated evidence.

Next steps

  1. Keep SMTP and policy-host certificate inventories separate.
  2. Reject validation-bypass evidence.
  3. Apply usage-specific DANE rules when relevant.

Founder / Non-technical

Assign endpoint ownership

Make the result useful to the person responsible for service continuity.

Next steps

  1. Assign ownership for each receiving TLS endpoint.
  2. Include failover paths in acceptance scope.
  3. Require evidence before broadening allowed MX patterns.

[ FAQs ]

Frequently asked questions

Is the policy website certificate the same as the SMTP certificate?

No. They protect different endpoints and require separate checks.

RFC 8461 §3.3 RFC 8461 §4.2

Does a matching MX pattern validate the certificate?

No. Matching the selected MX against a policy pattern and validating its certificate are separate conditions. MTA-STS SMTP requires a matching SAN DNS-ID; a generic Common Name-only hostname-check success is insufficient.

RFC 8461 §4.1 RFC 8461 §4.2

Which hostname matters in the example?

The receiving MX target mx1.example.net, not the visible recipient domain.

RFC 8461 §4.2

Can a backup MX cause intermittent reports?

Yes. A route used only during failover may present a different certificate.

Does the fixture validate X.509?

The new offline model checks exact SAN membership only. The retained historical loopback capture documents real hostname, expiry and trust checks under its disposable lab CA; it was not rerun here and is not a full MTA-STS compliance test.

Should hostname checks be disabled?

No. That would remove the condition being tested.

Are all DANE usages equivalent to STS name checks?

No. DANE authentication depends on the TLSA usage and its rules.

RFC 7672 §3.1.1

Can old reports describe a repaired certificate?

Yes. Read the observation period and compare it with the repair time.

RFC 8460 §4.1

[ Sources ]

Sources

  1. RFC 7672

    Sections §3.1.1

  2. RFC 8460

    Sections §4.1 · §4.3.1

  3. RFC 8461

    Sections §3.3 · §4.1 · §4.2

  4. OpenSSL hostname-check semantics

    Sections DESCRIPTION: SAN, CN and subject-check flags

[ Glossary ]

Glossary

SAN DNS-ID
A DNS identity in the certificate subject alternative name; MTA-STS SMTP validation requires a matching entry.

RFC 8461 §4.2

MX pattern
An allowed receiving-host pattern in the applied MTA-STS policy, separate from a certificate identity.

RFC 8461 §4.1

Reporting period
The observation interval reported, which can precede arrival of the report.

RFC 8460 §4.1

Enterprise provenance

Why top financial institutions and governments retired their email security dashboards.

Traditional email authentication platforms sell you a dashboard and ask your engineers to do the labour. In high-stakes environments, manual oversight means delayed enforcement, configuration drift and unmonitored shadow domains.

EmailAuth is the agentic email provenance platform. Our agents enforce p=reject, keep third-party senders aligned and move your domains to full RFC 9989 compliance, the new DMARC standard,, with human-in-the-loop auditability.

Governments Large enterprises Tier-1 financial institutions

SaaS · Dedicated tenancy · Air-gapped on-prem · BYOK encryption

Knowledge Base