Lakehouse

TL;DR A 2026 lakehouse has seven layers: object storage, open table format, catalog, compute engine, orchestration, transformation, and governance Apache Iceberg is the default table format; catalog choice (Unity, Polaris, Nessie) depends on your primary engine Databricks or Snowflake as compute, dbt or SQLMesh for transformation, orchestration via Dagster or Airflow Governance and observability are the layer most often skipped and most expensive to retrofit Default stack: ship end-to-end on a small slice first, then expand - do not spend six months evaluating before data flows The word “lakehouse” has been doing a lot of work for the last five years. It has been used to describe everything from a thin SQL layer over object storage to a fully integrated platform with governance, lineage, ML training, and BI built on top. Like most umbrella terms, this elasticity has been useful for marketers and confusing for engineers. ...

TL;DR Data pipelines were optimised for human consumers - dashboards, BI tools, analysts. In 2026 a growing share of pipeline output flows directly to language models, agents, and retrieval systems. That changes the design constraints in ways that catch teams off guard. Aggregation matters less. Context fidelity matters more. Freshness behaves differently. Schema moves from rigid to negotiated. Cost shifts from compute to tokens. The biggest mistake is treating an LLM consumer as if it were just another dashboard. It is not. It does not skim, it does not interpret charts, it does not have working memory across rows. It needs to be fed. The new patterns - retrieval-aware partitioning, embedding pipelines, structured-document outputs, prompt-shaped views, evaluation harnesses for data quality - are the actual subject of “AI-native data engineering” in 2026. The Underlying Shift For thirty years the implicit consumer of every data pipeline was a human looking at a screen. Even when the pipeline ended in an API or a CSV, the conceptual end-user was someone who would interpret the output with judgement, context, and skim-reading. ...

TL;DR The open table format war between Apache Iceberg, Delta Lake, and Apache Hudi is effectively over in 2026 - and the outcome is not a single winner but a clear settlement. Iceberg has won the role of the neutral standard that engines and platforms expect to read and write. It is the format you choose when you do not want to be coupled to a single vendor. Delta has won the role of the incumbent default inside the Databricks ecosystem and remains a strong choice if Databricks is your primary engine. Delta UniForm has narrowed the gap by letting Delta tables expose Iceberg metadata. Hudi has not won a category outright. It retains a smaller but loyal user base for streaming-heavy and CDC-heavy workloads, where its design choices still genuinely fit. The interesting battle has moved up the stack to the catalog layer. The format question is mostly settled. The catalog question is the new fight. The Format Wars - A Brief History For most of the early 2020s the lakehouse story was a three-way argument about how to put ACID transactions on top of object storage. ...

TL;DR With the open table format wars largely settled, the strategic fight in 2026 has moved up to the catalog layer - the system that manages tables, namespaces, governance, and access. Four credible open or open-ish catalogs are now in serious play: Unity Catalog (Databricks), Polaris (Snowflake), Apache Gravitino (Datastrato/community), and Project Nessie (Dremio/community). All four implement the Iceberg REST catalog spec to varying degrees, which means clients can talk to them through a common protocol. The differentiation has moved to governance, multi-tenancy, lineage, federation, and developer experience. Unity is the most production-mature and the most coupled to Databricks. Polaris is the cleanest open implementation of the REST spec. Gravitino is the most ambitious in scope - aiming to catalog non-table assets too. Nessie is the most opinionated about Git-style branching for data. The winning catalog will probably not be a single project. It will be the protocol (Iceberg REST) plus multiple compliant implementations plus federation between them. That is the picture 2026 ends with. Why The Catalog Layer Matters Now A table format defines how data is laid out on disk. A catalog defines: ...

TL;DR Apache Iceberg won the open table format war in 2026 - both Snowflake and Databricks treat it as first-class Iceberg is a table format (metadata on top of Parquet), not a storage format - ACID, time travel, and schema evolution live in the metadata tree Catalog choice (Polaris, Nessie, Unity) is now the harder decision than picking Iceberg itself Every stack layer above Iceberg is replaceable: change engine, catalog, or storage without rewriting data files For platform context, pair this with The Modern Lakehouse Stack In 2023, the question was “which open table format will survive - Iceberg, Delta, or Hudi?” In 2026, that debate is over. Apache Iceberg won, and it won for reasons that have almost nothing to do with its raw performance. ...

A New Era for Open Data Formats Snowflake has announced the general availability of Snowflake Storage for Apache Iceberg on both AWS and Azure, marking a significant shift in how enterprises can build open, interoperable data lakehouses. This development combines Snowflake’s enterprise reliability and governance capabilities with the flexibility and openness of Apache Iceberg, one of the most promising open table formats in the data ecosystem. For a deeper look at Iceberg itself, see Apache Iceberg in 2026, and for where this sits in the broader platform picture see The modern lakehouse stack. ...

TL;DR By 2026, Databricks and Snowflake have converged - both claim lakehouse status; the old binary is outdated Databricks wins on transformation, ML, and cost at scale; Snowflake wins on SQL simplicity and BI integration Choose on workload (analytics vs ETL vs ML), team skills (SQL-first vs code-first), budget, and existing ecosystem Iceberg support on both sides makes multi-engine portability real - the mistake is choosing on hype, not fit For stack context, see The Modern Lakehouse Stack The views in this post are my own personal reflections on the data industry, written in my own time. They are not about any specific employer, team, or colleague, past or present, and do not draw on any non-public information. ...

Overview Databricks offers certification tracks aligned to common roles: Data Engineer, Data Analyst, Apache Spark Developer, Machine Learning Engineer, and Generative AI Engineer. All certifications: Validity: 2 years from pass date Cost: $200 per exam attempt Format: Multiple choice, proctored online Recent Updates (2026): Emphasis on Lakeflow Declarative Pipelines (the evolution of DLT), Unity Catalog, liquid clustering, predictive optimization, AUTO CDC, Lakehouse Federation, and serverless compute Choose a certification based on your: ...

The views in this post are my own personal reflections on industry patterns, written in my own time. They are not about any specific employer, team, or colleague, past or present, and do not draw on any non-public information. TL;DR Unity Catalog is a unified access-control and metadata layer for tables, volumes, models, and notebooks - it is not a data-quality tool, a discovery engine, or a masking system, and teams expecting those will be disappointed Migrating from Hive metastore remains the biggest operational challenge in 2026; the hybrid path (migrate reference data first, stage the rest) is the most common in practice Design catalogs around medallion layers (bronze/silver/gold), not per-environment schema sprawl, and grant permissions only through roles, never directly to users Budget realistically: $50k-$200k of engineering time for large-organisation migrations, roughly a third to half of one engineer’s time ongoing, and under 5% query overhead Skip UC for single-team startups, sandbox data, and some streaming workloads; full adoption typically takes 6-12 months Unity Catalog sounds straightforward: “one governance layer for all your data and AI assets.” In theory, it’s elegant. In practice, you’ll run into gotchas that docs don’t prepare you for. ...