Governance

For most of the last decade, “ethical data use” was something that happened in a different building. The lawyers wrote the privacy policy, the data protection officer ran the impact assessment, and the engineers built whatever the ticket said. The ethics lived in a PDF, and the pipeline lived in the warehouse, and the two rarely met. In 2026 that separation has quietly collapsed. The reason is not that engineers suddenly became more principled - it is that the decisions which determine whether data is used ethically are now made at the schema, the table, and the access-control layer, and those are the engineer’s decisions. Consent, deletion, minimisation, provenance, bias: every one of them is now something you either build into the pipeline or fail to. This is a practical look at what that means. ...

TL;DR With the open table format wars largely settled, the strategic fight in 2026 has moved up to the catalog layer - the system that manages tables, namespaces, governance, and access. Four credible open or open-ish catalogs are now in serious play: Unity Catalog (Databricks), Polaris (Snowflake), Apache Gravitino (Datastrato/community), and Project Nessie (Dremio/community). All four implement the Iceberg REST catalog spec to varying degrees, which means clients can talk to them through a common protocol. The differentiation has moved to governance, multi-tenancy, lineage, federation, and developer experience. Unity is the most production-mature and the most coupled to Databricks. Polaris is the cleanest open implementation of the REST spec. Gravitino is the most ambitious in scope - aiming to catalog non-table assets too. Nessie is the most opinionated about Git-style branching for data. The winning catalog will probably not be a single project. It will be the protocol (Iceberg REST) plus multiple compliant implementations plus federation between them. That is the picture 2026 ends with. Why The Catalog Layer Matters Now A table format defines how data is laid out on disk. A catalog defines: ...

The views in this post are my own personal reflections on industry patterns, written in my own time. They are not about any specific employer, team, or colleague, past or present, and do not draw on any non-public information. TL;DR Unity Catalog is a unified access-control and metadata layer for tables, volumes, models, and notebooks - it is not a data-quality tool, a discovery engine, or a masking system, and teams expecting those will be disappointed Migrating from Hive metastore remains the biggest operational challenge in 2026; the hybrid path (migrate reference data first, stage the rest) is the most common in practice Design catalogs around medallion layers (bronze/silver/gold), not per-environment schema sprawl, and grant permissions only through roles, never directly to users Budget realistically: $50k-$200k of engineering time for large-organisation migrations, roughly a third to half of one engineer’s time ongoing, and under 5% query overhead Skip UC for single-team startups, sandbox data, and some streaming workloads; full adoption typically takes 6-12 months Unity Catalog sounds straightforward: “one governance layer for all your data and AI assets.” In theory, it’s elegant. In practice, you’ll run into gotchas that docs don’t prepare you for. ...

This is the set of re:Invent 2022 slide decks I found most useful when they were published, grouped by topic. Each entry links to the official AWS-hosted PDF and carries a short, plain-language note about what the session is useful for in practice - so you can decide which decks are worth reading before committing the time. For the full session video recordings, see the AWS Events channel on YouTube. DevOps Amazon’s approach to high-availability deployment The practices Amazon’s own delivery teams use to reach near-zero deployment failure rates. Most of the value is in the guardrail patterns - pre-production gates, automated rollback triggers, and how to design a release process that protects itself from human error. ...