Lakeflow

TL;DR Lakeflow Declarative Pipelines is the evolution of Delta Live Tables, and the rename signals a real shift in mental model: from “tables and dependencies” to “data flows and transformations” The three core building blocks are streaming tables (incremental, append-only), materialized views (full recompute, best for aggregations), and AUTO CDC for slowly-changing dimensions without hand-rolled merge logic Physical optimisation is increasingly automatic in 2026 - liquid clustering is the default, predictive optimization handles maintenance, and Z-order is legacy Keep hand-rolled Spark jobs for imperative logic, external API calls, and ML workloads; Lakeflow is for SQL-shaped data movement Lakeflow and dbt are complementary rather than competitors - some teams use Lakeflow for ingestion to silver and dbt for silver-to-gold If you’ve been writing Delta Live Tables (DLT) pipelines, you’ve been building with Lakeflow without knowing the new name. In 2026, the rebranding matters because it signals how Databricks now wants you to think about declarative pipeline design. ...

The 2026 Databricks Baseline Databricks in 2026 looks much more opinionated than it did just a few years ago. For most new data engineering work, the default stack is now clear: Unity Catalog for governance managed tables where possible serverless compute for notebooks, SQL, pipelines, and jobs Lakeflow Declarative Pipelines for batch and streaming data products liquid clustering instead of old-style partition design for many workloads That shift matters because the platform has moved beyond “bring your own clusters and tune everything manually.” The modern Databricks approach is increasingly declarative, governed, and automated. ...