API ingestion strategies?
retained text
I have to get a lot of JIRA data from the API, issues, projects, users, permissions etc. How would you handle this? I currently save it raw in the bronze and derive silver tables from it using SCD2, but what if an api fails? The silver would have invalid data. Its a simple task, but somehow I am lost (new from university) Does it even make sense to use SCD2 here?
document fetched 2026-09-18 00:24 · canonical https://reddit.com/r/dataengineering/comments/1wiv4fs/api_ingestion_strategies