[ Business value ]

Assess reporting quality using the same evidence across tools

Compare reporting tools on identical raw inputs and explicit count definitions.

By the EmailAuth research team · 3 min read

[ Key points ]

  • Compare reporting tools on identical raw inputs and explicit count definitions.
  • Reconcile duplicate reports, conflicts and policy-domain filters before comparing percentages.
On this page

Fix the evidence population

A dashboard percentage has little meaning without its denominator. Define the reporter set, time windows, unique report identity rules and messages represented by each row. Compare the same immutable input files across tools. Record import settings and supported schema versions so differences in parsing can be investigated rather than dismissed as presentation choices.

Preserve raw authentication results separately from policy-evaluated alignment. A report can show an SPF pass for an unrelated envelope domain while the policy-evaluated SPF alignment fails. Aggregate row counts represent the reported message population, not necessarily every message the organization sent. Reporting participation and delayed arrivals limit coverage, as described by RFC 9990.

Reconcile duplicates and row totals

Consider a written normalized fictional report example, not an XML dataset. Reporter A contributes a report with eighty aligned and twenty failing messages. An exact repeated import of that same report must not double the denominator. Reporter B contributes fifty aligned messages under the same textual report ID; its different reporter identity prevents inappropriate deduplication in this defensive test policy. RFC 9990 Section 3.5.1 requires report identifiers to be unique among reports to the same domain; this intentional collision is a robustness case, not an endorsement of conforming identifier reuse. The expected unique total is one hundred fifty, with one hundred thirty aligned.

A conflicting copy using the same reporter, report ID, window and policy domain but different rows is quarantined for review rather than silently replacing the first import. A future implementation should compare canonical payloads under a documented normalization rule to detect this conflict. These choices are a proposed ingestion policy; they do not claim that report IDs alone are globally unique or that every supplier implements the same rules.

Explain differences before judging quality

If a tool displays a different total, trace the discrepancy to a row, report or documented exclusion. Check time-zone boundaries, multiple reporting domains, parse errors, repeated imports and filters. A high pass percentage can result from discarding malformed or unknown rows. A lower percentage may simply include a wider population. Neither result establishes quality without reconciliation.

Require an export linking the displayed result to source report identities, counts and classification rules. The written example demonstrates normalized accounting and duplicate handling only; it does not validate compression, XML security, schema parsing or a production reporting service. Those need separate tests with approved raw files. Finish with an exception register that makes unsupported formats and uncertain attribution visible instead of hiding them in a successful-import count.

Inspect the scoped case ledger

The revised ingestion key includes reporter, report ID, report window and policy domain. The first report for brand.test is included; its identical copy is excluded; the second reporter's report is included; the conflicting first-reporter copy is quarantined. This preserves the brand.test population at 150 messages, 130 aligned.

A fifth control uses the first reporter's identical report ID and window but policy domain other.test, with seven aligned messages. It is included as a separate report, not mistaken for a duplicate or conflict. The combined total is 157, while the brand.test dashboard still uses 150. The written case ledger states each hypothetical input and intended decision. This proposed key and conflict policy must be assessed against the actual formats and populations being imported; it is not a universal claim about every reporting tool.

Apply the correct denominator

For brand.test the illustrative calculation is (80 + 50) / (80 + 20 + 50) = 130 / 150, or about 86.67 percent. Including other.test changes the combined calculation to 137 / 157, about 87.26 percent, and answers a different question. Quarantined conflicting data must remain visible as unresolved, not silently converted to a good or bad message count. These are arithmetic demonstrations over invented labels, not parsed reports or observed product results.

Protocol scope and evaluation limits

RFC 9989 Sections 4.4, 5.3.5 and 5.4 distinguish authenticated identifiers, alignment and receiver handling. A valid signature alone is not a complete DMARC verdict, and a pass does not validate business content. RFC 9990 Sections 3.1, 3.5.1, 3.5.4 and 7 describe aggregate evidence, report identity, duplicates and privacy. Proposed commercial evaluation, staffing and authority gates here are editorial recommendations, not RFC-mandated product features. No absent executable supplement is required to read these written examples; operational claims require separately authorized observed tests.

[ Takeaway ]

Reconcile duplicate reports, conflicts and policy-domain filters before comparing percentages.

[ By role ]

What this means for you

Choose your role for practical next steps.

IT / Mail Admin

Preserve reproducibility

Record the exact inputs and observations needed to repeat this specific evaluation.

Next steps

  1. Reconcile exact duplicates and quarantine conflicting report copies.
  2. Keep raw evidence separate from interpretation and labels.
  3. Verify any live change through the authorized change process.

CISO / SecOps

Control authority and uncertainty

Prevent incomplete observations from turning into unsupported security or compliance conclusions.

Next steps

  1. Check raw report retention, export access and ingestion security separately.
  2. Require explicit approval for consequential changes.
  3. Retain unknowns and negative controls in the final decision record.

Founder / Non-technical

Own the decision

Keep the evaluation tied to the business task and the limits of the evidence.

Next steps

  1. Approve the report population and denominator used in comparisons.
  2. Assign an accountable owner for unresolved cases.
  3. Review retained workload and exit requirements before committing.

[ FAQs ]

Frequently asked questions

What is the brand.test example denominator?

150 messages across two unique hypothetical reports; the separate other.test population is excluded from this domain view.

How many are aligned?

130 for brand.test under the invented normalized labels; including other.test yields 137 out of 157.

Should an exact repeated import count twice?

No, under this explicitly defined deduplication policy.

Is report ID globally unique?

Do not assume so. Include reporter and relevant context.

What happens to conflicting copies?

Quarantine and investigate rather than silently overwrite.

Does this example parse XML?

No. It is written accounting over hypothetical normalized data, not an executed parser or production test.

Can reporting cover every sent message?

Not necessarily. Reporting participation and receipt limit coverage.

What proves a displayed percentage?

A traceable numerator, denominator, input population and classification rule.

[ Sources ]

Sources

  1. RFC 9990

    Sections §3.1 · §3.5.1 · §3.5.4 · §7

  2. RFC 9989

    Sections §4.4 · §4.7 · §4.10.1 · §5.3.5 · §5.4

[ Glossary ]

Glossary

Alignment
The relationship between an authenticated domain and the visible From domain under the applicable DMARC rules.
Evidence population
The explicitly defined set of observations included in an analysis.
Negative control
A deliberately failing or prohibited case used to check that a test detects the intended problem.

Enterprise provenance

Why top financial institutions and governments retired their email security dashboards.

Traditional email authentication platforms sell you a dashboard and ask your engineers to do the labour. In high-stakes environments, manual oversight means delayed enforcement, configuration drift and unmonitored shadow domains.

EmailAuth is the agentic email provenance platform. Our agents enforce p=reject, keep third-party senders aligned and move your domains to full RFC 9989 compliance, the new DMARC standard,, with human-in-the-loop auditability.

Governments Large enterprises Tier-1 financial institutions

SaaS · Dedicated tenancy · Air-gapped on-prem · BYOK encryption

Knowledge Base