Data Engineering on Databricks
Building pipelines that other people depend on — ingestion, Delta tables, incremental loads, and the tests that stop a bad batch reaching a dashboard.
A pipeline is easy to build once and hard to keep correct. This course spends most of its time on the second problem: what happens when the source changes shape, when yesterday's file arrives today, and when someone needs the numbers restated.
Everything is taught on Databricks, which is the platform this lab builds on and partners with.
Every lesson is free and needs no sign-up.
Syllabus
What a lakehouse is
Files, tables, and why Delta exists — told as the problem it solves rather than the architecture diagram.
Getting data in
Batch files, streaming sources, and Auto Loader on a folder that keeps growing.
Transformation
Bronze, silver and gold as a working habit, not a slogan.
Incremental loads
Merges, late-arriving rows, and reprocessing a day without reprocessing a year.
Orchestration
Jobs, dependencies, retries, and what should happen at 3am when a task fails.
Quality and governance
Expectations on a pipeline, and using Unity Catalog so people can find a table without asking you.