# Getting started with Apache Spark Declarative Pipelines (SDP) tutorial for Open Source and Lakeflow — retained record

> Retained reddit record from reddit:dataengineering.

Canonical: https://highsignal.app/data/records/d7f72c00c10cdc54

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:dataengineering)  published 2026-09-21 07:42 ingested 2026-09-22 02:36

[original source ↗](https://www.reddit.com/r/dataengineering/comments/1wm64sf/getting_started_with_apache_spark_declarative/)

## retained text



```
A while ago I put this SDP tutorial together after presenting at major European and APAC conferences. Those conferences are quite technical: you have to submit a proposal to the CFP, and they have a rejection rate of \~90%. I presented at Devoxx and DataEngBytes.  


At those conferences I did some live coding with Spark Declarative Pipelines, mostly on OSS with VS Code and PySpark, but I also showed SDP with the Lakeflow Free Edition.

Since I got a lot of requests, I decided to share the content along with the Git repos here. I posted it on Databricks for visibility, but it comes with 2 tutorials covering the same use case, "streaming data from every plane currently in the air", and one of the tutorials is 100% open source.

TL;DR, use this Spark Declarative Pipelines tutorial if you...

* enjoy working with code (SQL and Python)
* are looking for a compelling Getting Started with Apache SDP example with just a few lines of code
* want a streaming example with real-time IoT data from events happening right now
* enjoy using a custom PySpark data source "opensky" – maybe even in your next Spark streaming demo :)
* don't want to spend money: use OSS or the Databricks Free Edition

Here it is:  
[How to get started with Spark Declarative Pipelines (SDP) tutorial (Open Source or Lakeflow)](https://www.databricks.com/discover/how-to-get-started-with-spark-declarative-pipelines)
```



document fetched 2026-09-22 00:23 · canonical https://reddit.com/r/dataengineering/comments/1wm64sf/getting_started_with_apache_spark_declarative
