← sources

API ingestion strategies?

source: reddit (reddit:dataengineering)published 2026-09-17 14:11ingested 2026-09-18 02:38

retained text

I have to get a lot of JIRA data from the API, issues, projects, users, permissions etc.

  
How would you handle this? I currently save it raw in the bronze and derive silver tables from it using SCD2, but what if an api fails? The silver would have invalid data.

Its a simple task, but somehow I am lost (new from university)

Does it even make sense to use SCD2 here? 

document fetched 2026-09-18 00:24 · canonical https://reddit.com/r/dataengineering/comments/1wiv4fs/api_ingestion_strategies