How would you detect changes in thousands of provider XML feeds without treating every job as new?
retained text
Design question I've been thinking about. Setup: external providers publish XML feeds of job vacancies to S3. Each feed is the provider's full current snapshot, not a list of changes. Schemas differ per provider, some fields are optional, and feeds can hold 100k+ jobs. Each run needs to work out what's new, what's updated and what's gone, without rewriting the other \~98% of unchanged jobs or spamming downstream services. The parts I find interesting: 1. Provider timestamps like updated\_date aren't reliable, so how do you detect a real change? 2. "Missing from today's feed" usually means deleted, but a truncated or empty feed looks identical. How do you tell the difference before you take live jobs offline? 3. If a run dies at 60k of 100k records, how do you rerun without duplicates, repeat notifications or a half-old, half-new dataset? What approaches have you used? Hashing records, staging tables, soft deletes with a grace period, deletion thresholds, something else? Curious what has held up in production and what hasn't
document fetched 2026-10-14 00:37 · canonical https://reddit.com/r/aws/comments/1x4jz8j/how_would_you_detect_changes_in_thousands_of