# How would you detect changes in thousands of provider XML feeds without treating every job as new? — retained record

> Retained reddit record from reddit:aws.

Canonical: https://highsignal.app/data/records/1cf70b6245b8cba0

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:aws)  published 2026-10-13 01:39 ingested 2026-10-14 02:42

[original source ↗](https://www.reddit.com/r/aws/comments/1x4jz8j/how_would_you_detect_changes_in_thousands_of/)

## retained text



```
Design question I've been thinking about.

Setup: external providers publish XML feeds of job vacancies to S3. Each feed is the provider's full current snapshot, not a list of changes. Schemas differ per provider, some fields are optional, and feeds can hold 100k+ jobs.

Each run needs to work out what's new, what's updated and what's gone, without rewriting the other \~98% of unchanged jobs or spamming downstream services.

The parts I find interesting:

1. Provider timestamps like updated\_date aren't reliable, so how do you detect a real change?  
2. "Missing from today's feed" usually means deleted, but a truncated or empty feed looks identical. How do you tell the difference before you take live jobs offline?  
3. If a run dies at 60k of 100k records, how do you rerun without duplicates, repeat notifications or a half-old, half-new dataset?

What approaches have you used? Hashing records, staging tables, soft deletes with a grace period, deletion thresholds, something else? Curious what has held up in production and what hasn't
```



document fetched 2026-10-14 00:37 · canonical https://reddit.com/r/aws/comments/1x4jz8j/how_would_you_detect_changes_in_thousands_of
