Getting started with Apache Spark Declarative Pipelines (SDP) tutorial for Open Source and Lakeflow
retained text
A while ago I put this SDP tutorial together after presenting at major European and APAC conferences. Those conferences are quite technical: you have to submit a proposal to the CFP, and they have a rejection rate of \~90%. I presented at Devoxx and DataEngBytes. At those conferences I did some live coding with Spark Declarative Pipelines, mostly on OSS with VS Code and PySpark, but I also showed SDP with the Lakeflow Free Edition. Since I got a lot of requests, I decided to share the content along with the Git repos here. I posted it on Databricks for visibility, but it comes with 2 tutorials covering the same use case, "streaming data from every plane currently in the air", and one of the tutorials is 100% open source. TL;DR, use this Spark Declarative Pipelines tutorial if you... * enjoy working with code (SQL and Python) * are looking for a compelling Getting Started with Apache SDP example with just a few lines of code * want a streaming example with real-time IoT data from events happening right now * enjoy using a custom PySpark data source "opensky" – maybe even in your next Spark streaming demo :) * don't want to spend money: use OSS or the Databricks Free Edition Here it is: [How to get started with Spark Declarative Pipelines (SDP) tutorial (Open Source or Lakeflow)](https://www.databricks.com/discover/how-to-get-started-with-spark-declarative-pipelines)
document fetched 2026-09-22 00:23 · canonical https://reddit.com/r/dataengineering/comments/1wm64sf/getting_started_with_apache_spark_declarative