Repeated report delivery should not inflate traffic metrics. At the same time, similar reports can represent distinct observations. A safe deduplication process identifies proven duplicates, preserves raw evidence and isolates uncertain collisions rather than deleting anything that looks familiar.
Preserve evidence before normalization
References: RFC 9990 §3.5.1, RFC 9990 §3.5.4.
Store the original protected attachment, its cryptographic fingerprint, arrival context and parser version. Maintain a separate import decision record. Raw bytes and normalized report data serve different purposes: two attachments can encode equivalent content while differing in compression or harmless formatting.
RFC 9990 defines report identifiers to help recognize duplicates and requires reuse of identifying information on retransmission. A filename or subject alone is nevertheless untrusted input. Validate the parsed report and compare its policy domain, reporter, interval and content before making an irreversible data-loss decision.
Distinguish duplicates from collisions
An identical retained attachment from the same report context is a strong candidate for idempotent handling. A matching report identity with different content is a conflict, not automatically a safe duplicate. Preserve both versions and mark the affected metric contribution provisional and apply an explicit whole-report inclusion policy until the relationship is resolved.
Similar row values are not enough. Two valid reports may independently observe the same source IP and authentication results. Deduplicating rows globally can erase legitimate repeated traffic. Keep report-level provenance attached to every row.
Use a reversible import decision
A practical implementation assigns a stable internal report record and records every arrival separately. A confirmed repeat points to the already counted report instead of inserting its rows again. Log the match criteria and algorithm version so a later repair can reconstruct what happened.
Illustrative test set: file A contains 100 observations; A arrives twice. The counted total remains 100 and arrivals become two. File B uses the same report identity but contains 120 observations. The system flags a content conflict rather than silently replacing 100 with 120 or counting 220. File C is a verified distinct report with 100 similar observations and contributes separately. These fixtures are proposed tests, not production results.
Treat overlap as a different problem
References: RFC 9990 §3.5.4.
Partial interval overlap does not reveal which individual observations repeat. RFC 9990 section 3.5.4 recognizes that the impact cannot be cleanly separated from aggregate information alone. Do not subtract a time-proportional fraction and describe the remainder as exact.
Keep uncertain reports outside the primary reconciled total or apply another documented whole-report policy, preserving the excluded material and its impact. Publish the policy with the metric. Test replay, content conflicts and distinct same-looking reports before calling the importer idempotent.
Audit the import decision log
Every excluded arrival should point to the retained counted report and the comparison that justified exclusion. Every unresolved collision should have an owner and a visible status. Restrict permissions to change those decisions, and keep a revision history when a parser defect requires reimporting old evidence. Test recovery by rebuilding a small metric from raw reports and the saved decision log. If the reconstruction differs, investigate before using the totals for a policy decision. Idempotence must survive retries and restarts, not only a clean first import.
Continue the investigation
Use the aggregate-report walkthrough to interpret the underlying fields and the alignment diagnosis guide when a raw authentication pass does not satisfy DMARC.
Executed synthetic evidence
The supplied locally rerun offline Python fixture defines its logical key as (validated reporter identity, policy domain, report_id). It compares a simplified (start, end, count) tuple separately from the logical key; full-payload hashing and normalization are outside this model. Validating the reporter is an input prerequisite, not something a self-reported name proves. The small demonstration accepts only supplied trusted identities and does not implement a production authentication boundary.
Its executed tests produce the following outcomes:
- Case: First report; Input distinction: receiver-a.example, brand.example, R1, count 100; Actual result: accepted; Additional counted observations: 100
- Case: Exact replay; Input distinction: Same key and payload; Actual result: duplicate; Additional counted observations: 0
- Case: Content collision; Input distinction: Same key, count changed to 120; Actual result: content_conflict_hold; Additional counted observations: 0
- Case: Same ID, different source; Input distinction: receiver-b.example, brand.example, R1, count 100; Actual result: accepted; Additional counted observations: 100
- Case: Partial overlap; Input distinction: receiver-a.example, R2, interval 06:00–18:00 after 00:00–12:00; Actual result: overlap_hold; Additional counted observations: 0
Assertions verify all five outcomes and a counted total of 200. The conflict and overlap inputs remain retained for review. The illustrative rule keeps the first admitted report counted and holds the later conflicting arrival; it does not silently replace the earlier report or claim that it is necessarily correct. A production system should expose the affected earlier count as provisional and allow an audited correction. The same-ID/different-source input deliberately tests an identifier collision; it is not a claim that ID reuse meets RFC 9990’s uniqueness requirement. Reporter context prevents automatic collapse of potentially independent observations while the collision is investigated. This run verifies the fixture logic, not database concurrency, restarts, source authentication or schema completeness.
Keep the result within the tested scope
The supplied local demonstration verifies only its small synthetic parser, supplied stage inputs and calculations. No real DNS authorization, SMTP delivery, collector migration, provider participation or production database behavior was tested. Original inputs remain archived, and any live comparison needs its own protected evidence and accountable approval.