Incremental synchronization should be able to stop halfway through a page and restart without losing evidence. The difficult case is not the first successful download. It is the timeout after storage, the duplicate delivery after restart and the report that arrives after the apparent reporting period has ended.
Define the unit of identity
Use the provider's documented report identity and retain its tenant and reporter context. Report IDs should not be assumed globally unique across customers or sources. Store a source payload fingerprint alongside the identity. An identical repeat can be treated as a duplicate; the same identity with different bytes should be investigated or handled through a documented revision model.
RFC 9990 provides the structure and semantics of aggregate reports. Preserve original reporting periods and receipt times separately. A collector's incremental watermark should not silently reinterpret a historical evaluation period as an arrival timestamp.
Commit data and progress together
Fetch a bounded page, validate scope and schema, write the normalized data with its provenance and only then advance the checkpoint. Prefer a transaction that commits the data and progress atomically. If that is unavailable, design for replay with a stable uniqueness constraint. A cursor saved before the records are durable creates a loss window.
Suppose a collector stores page one durably but crashes before advancing its progress marker. On restart, it may fetch the page again. A properly scoped stable identity constraint should avoid adding exact repeats, while allowing the next page to be committed. If the progress marker had advanced before the records were durable, restart could skip evidence entirely. These are hypothetical failure states, not an executed file or database test.
If a repeated identity arrives with different source bytes, preserve the original and the conflict for a documented revision decision. A changed hash is a signal to investigate, not proof that the message counts changed: formatting or normalization can also affect bytes. Test the actual storage transaction, crash recovery and source-byte comparison under the approved implementation; this guide asserts no durability guarantee.
Handle lateness without claiming certainty
Use a bounded overlap window when the actual API supports re-querying arrivals, then deduplicate by identity. Choose the overlap using observed ingestion delays and operational needs. A finite overlap cannot guarantee discovery of arbitrarily late data. Add periodic full reconciliation or another documented backfill mechanism and record the remaining coverage limitation.
Keep amendments distinct from exact duplicates. Retain both hashes on conflict and prevent an unexplained replacement from silently changing an already-issued customer report. Recompute affected summaries under a new revision when a valid correction is accepted, with a trace back to the previous result.
Recover under the real service contract
Respect rate limits, Retry-After and version-specific pagination. GitHub's API best practices offer a concrete example of bounded, considerate request behavior; they do not define an authentication vendor's quotas. Treat a 429 or transient failure as a reason to pause and retry within policy, not as permission for unlimited parallel requests.
Monitor last successful checkpoint, oldest unprocessed arrival, retry exhaustion and conflicts. A healthy API response with a stuck cursor is still a failed synchronization. On an ambiguous outcome, inspect stored identities and progress before retrying a side-effecting step. Preserve source bytes and the previous checkpoint so recovery is evidence-led. No API call, retry simulation or EmailAuth synchronization capability is demonstrated here. GitHub’s rate-limit documentation notes that it may return 403 as well as 429 for rate limits; distinguish documented limit signals from permanent authorization denial rather than assigning one meaning to every status.