← sources

Getting started with Apache Spark Declarative Pipelines (SDP) tutorial for Open Source and Lakeflow

source: reddit (reddit:dataengineering)published 2026-09-21 07:42ingested 2026-09-22 02:36

retained text

A while ago I put this SDP tutorial together after presenting at major European and APAC conferences. Those conferences are quite technical: you have to submit a proposal to the CFP, and they have a rejection rate of \~90%. I presented at Devoxx and DataEngBytes.  


At those conferences I did some live coding with Spark Declarative Pipelines, mostly on OSS with VS Code and PySpark, but I also showed SDP with the Lakeflow Free Edition.

Since I got a lot of requests, I decided to share the content along with the Git repos here. I posted it on Databricks for visibility, but it comes with 2 tutorials covering the same use case, "streaming data from every plane currently in the air", and one of the tutorials is 100% open source.

TL;DR, use this Spark Declarative Pipelines tutorial if you...

* enjoy working with code (SQL and Python)
* are looking for a compelling Getting Started with Apache SDP example with just a few lines of code
* want a streaming example with real-time IoT data from events happening right now
* enjoy using a custom PySpark data source "opensky" – maybe even in your next Spark streaming demo :)
* don't want to spend money: use OSS or the Databricks Free Edition

Here it is:  
[How to get started with Spark Declarative Pipelines (SDP) tutorial (Open Source or Lakeflow)](https://www.databricks.com/discover/how-to-get-started-with-spark-declarative-pipelines)

document fetched 2026-09-22 00:23 · canonical https://reddit.com/r/dataengineering/comments/1wm64sf/getting_started_with_apache_spark_declarative