Deduplicate report ingestion without losing corrections, policy scope or the difference between observation time and arrival time.
Treat retransmission as an ingestion problem
References: RFC 8460 §4.1, RFC 8460 §5.5.
A report may reach a collector more than once through retries, forwarding or multiple destinations. Counting each arrival as new telemetry inflates traffic and failure totals. Preserve the raw payload and a digest before normalizing it. Record arrival timestamp and intake channel separately from the report’s date-range. Reports normally cover a UTC day and may arrive later, so arrival-day grouping can shift incidents into the wrong operational window.
Design a key with explicit assumptions
References: RFC 8460 §3, RFC 8460 §4.4.
RFC 8460 defines report-id and reporting metadata, but it does not prescribe a universal collector deduplication algorithm. A practical local design uses authenticated reporter identity, report-id and reporting period as an identity key, plus a content digest to detect exact duplicate bytes or equivalent normalized content. The free-text organization-name alone is not a verified identity. Preserve the original document and authentication context so an administrator can investigate collisions rather than silently accepting a self-asserted organization label.
Distinguish duplicates from conflicting revisions
The synthetic fixture receives report R1 twice with the same identity key and normalized digest. It retains one analytical copy and records two arrivals. It then receives R1 with the same identity key but a different failure count and digest. That becomes a conflict requiring review, not an automatic additional report and not an overwrite of the original. A fourth arrival, R2 in a different period, remains separate. These rules are a documented local ingestion choice, not requirements imposed on all TLS-RPT collectors.
Keep policy boundaries after deduplication
References: RFC 8460 §4, RFC 8460 §4.4, RFC 8460 §7.
One report can contain multiple policies. Preserve policy-type, policy-domain, policy-string and mx-host when storing entries so a report-level duplicate decision does not flatten meaningful policy differences. Failure categories may overlap within an entry; deduplication does not make their detail counts additive. Reject or quarantine malformed records according to a documented ingestion policy, retaining safe diagnostic metadata. Do not repair a bad count by guessing or throw away the only raw evidence of a parsing problem.
Make replay safe and auditable
Use a deterministic normalization rule and version it. The supplied teaching fixture sorts JSON object keys before hashing; it does not define universal semantic equivalence for every JSON encoding or schema. Reject duplicate object keys and malformed data before normalization under a documented intake policy, so a parser cannot silently erase conflicting values. Avoid including ingestion timestamps in the content digest used for duplicate comparison. Store duplicate arrival events without adding their session counts again. For a replay after a parser fix, ensure the same report identity cannot increment totals twice. Test exact duplicates, changed payloads, reordered JSON keys and distinct periods. Access controls and retention limits still apply to the preserved operational data. The newly rerun local fixture demonstrates those idempotency cases; it does not implement a production database, verify a real reporter or claim that all corrections share one reporting convention.
Retained historical evidence and current checks
The transport observations below were captured on 7 November 2026 and inspected, not rerun as network tests, for this review. The new offline model checks are recorded separately.
The new offline ingest fixture stores every arrival in a separate list, records new, duplicate or conflict classification, and asserts two arrivals after the duplicate replay. The second arrival does not add an analytical report. A changed payload creates a conflict while the original digest is retained. The retained historical annotated report separately links a loopback hostname-validation failure to report-shaped data. No production database, authenticated intake service or real reporter was exercised. The historical transport runner requires Python with cryptography and loopback sockets. Its outcomes are bounded lab evidence, not current service health or production interoperability.
Primary-source basis
The field definitions and reporting limits come from RFC 8460, SMTP TLS Reporting, sections 3 and 4. The operational recommendations are distinguished from protocol requirements.