Inherited a PB scale data mess. Looking for tech & people strategies.
retained text
Hello good folks of /r/dataengineering, I need opinion &/ guidance on how you would handle such a situation. Inherited a large data data platform PB level, with a lot of pipelines, tables and tools. Built mostly on OSS and K8s stack. 1. Company grew very fast (think non-tech startup, public now), BigQuery is the primary warehouse, with a mix of batch and streaming. 2. Engineers came and went, lot of the platform is not maintained and people do not wish to touch what works. Eg. There are teams that still use Airflow 1.10. 3. There are a lot of pipelines and data sets are that are no longer in use, they're running on schedule and cost $$. 4. Schedules are all over the place. I found a pipeline that runs daily, powering a report that only gets sent out every two weeks. I was planning on mapping everything first then use that data to make culling decisions, but there is internal resistance from business. Folks are not even ready to spend time to add metadata on their own datasets. What do you folks recommend I do? Any suggestions on both technical and people strategies would be greatly appreciated. Please don't plug your tool here.
document fetched 2026-11-19 00:24 · canonical https://reddit.com/r/dataengineering/comments/1xzcbjq/inherited_a_pb_scale_data_mess_looking_for_tech