← sources

How would you detect changes in thousands of provider XML feeds without treating every job as new?

source: reddit (reddit:aws)published 2026-10-13 01:39ingested 2026-10-14 02:42

retained text

Design question I've been thinking about.

Setup: external providers publish XML feeds of job vacancies to S3. Each feed is the provider's full current snapshot, not a list of changes. Schemas differ per provider, some fields are optional, and feeds can hold 100k+ jobs.

Each run needs to work out what's new, what's updated and what's gone, without rewriting the other \~98% of unchanged jobs or spamming downstream services.

The parts I find interesting:

1. Provider timestamps like updated\_date aren't reliable, so how do you detect a real change?  
2. "Missing from today's feed" usually means deleted, but a truncated or empty feed looks identical. How do you tell the difference before you take live jobs offline?  
3. If a run dies at 60k of 100k records, how do you rerun without duplicates, repeat notifications or a half-old, half-new dataset?

What approaches have you used? Hashing records, staging tables, soft deletes with a grace period, deletion thresholds, something else? Curious what has held up in production and what hasn't

document fetched 2026-10-14 00:37 · canonical https://reddit.com/r/aws/comments/1x4jz8j/how_would_you_detect_changes_in_thousands_of