Analysing and serving · Chapter 17

The Five Pillars of Modern Analytics

Databricks publishes a framework called the five pillars of modern analytics. It is a useful map, we use it in architecture conversations, and this chapter walks through it with our own commentary attached.

The five pillars #

These are Databricks' names for them, in their order, with what each one actually asks of an organisation.

Pillar 1. Lakehouse architecture, the right foundation for modern analytics #

Open storage formats on cloud object storage, with Unity Catalog as the governance engine over everything: tables, schemas, files, dashboards, notebooks, models and pipelines. Object storage becomes the system of record; Delta Lake and Iceberg supply performance, reliability, metadata and schema evolution. Lakehouse Federation extends the same governance over external sources without replicating them.

What it asks of you is that the governance model comes first rather than last, which is the single most common sequencing mistake we are called in to fix. Unity Catalog and Tables and Storage are the working chapters.

Pillar 2. Data engineering and preparation, seamless analytics with Lakeflow #

One product covering ingestion, transformation and orchestration instead of a patchwork of ingestion tools, hand-coded batch scripts, ad hoc streaming jobs and a separate scheduler. Lakeflow Connect gives point-and-click ingestion from operational databases and SaaS applications with change data capture. Spark Declarative Pipelines let teams express pipelines in SQL or Python while the engine handles dependencies, incremental processing and recovery. Jobs handles scheduling and monitoring. Lakeflow Designer adds a no-code visual and natural language builder aimed at the last mile, where analysts would otherwise create ungoverned shadow pipelines.

The claim we find most defensible here is the declarative one: describe the end state and let the platform work out scheduling, scaling and recovery. Pipelines covers it, and Real-Time and Change Data Capture covers the connectors.

Pillar 3. Unified semantics, consistent logic for every analytics experience #

This is the pillar that matters most and the one most organisations have not done.

The problem it names is that every BI tool ships its own semantic layer with its own modelling language, so a metric gets defined one way in a dashboard tool, another way in a SQL script, and a third way inside an AI prompt. As teams scale, those definitions drift, and the guide identifies the absence of a shared semantic model as a primary cause of dashboard sprawl, poor AI accuracy and cross-team confusion.

The answer is Unity Catalog Business Semantics, which has two parts. Metric views are SQL-defined business measures and dimensions that live in the catalog, inherit its governance, and can be read by any engine that speaks SQL, rather than being locked inside LookML or DAX. Agent metadata is the contextual knowledge an AI agent needs to interpret a question correctly.

Pillar 4. AI-powered dashboards, modern BI built for the AI era #

AI/BI Dashboards run in place on the lakehouse against governed data and shared semantics, which removes extracts. They read metric views, so every dashboard works from the same definitions, and the materialised structure of those views can improve performance on common aggregations. Authoring is AI-assisted, and there is no per-seat licence limit shaping who is allowed to see data.

Reporting, Semantics, and Power BI is the working chapter, including the honest comparison against Power BI.

Pillar 5. Conversational agents, self-service insights with AI/BI Genie #

Genie is a conversational analytics agent grounded in Unity Catalog rather than a general chatbot pointed at a database. Each Genie space is scoped to a subject area and powered by Business Semantics, so metric views supply the structured definitions it reasons with. A knowledge store lets authors curate table and column descriptions, synonyms, sampled values, value dictionaries and SQL instructions. Feedback is captured, and good answers can become reusable knowledge snippets.

Where we agree, and where we would push #

We use this framework because the third pillar is correct and underrated, and because the ordering is right: each pillar genuinely depends on the ones before it. An organisation that buys the dashboards without doing the semantics gets prettier disagreement.

Three places we would push on it.

Pillar five is where the risk lives, and the guide is light on that. A conversational interface over governed data is genuinely useful and it will confidently answer a question about data it does not have. We would not put Genie in front of a business audience without a fixed set of questions with known answers, run as a test suite when the underlying tables change, with the pass rate published to the people relying on it. Reporting and Agents both cover this.

The framework is about analytics, and analytics is the easier half. Reading data and showing it to a person is a fundamentally lower-risk activity than reading data and acting on it. Everything in Governed Actions is outside this framework's scope, and for our clients it is where the hard problems are.

"Modern BI is broken because it was not designed for AI" is half the story. Modern BI was also broken for humans, for the ordinary reason that definitions lived in tools rather than in the platform, and that was true well before anyone put a language model in front of it. We prefer making the argument that way, because it does not depend on the reader being excited about AI to be persuasive.

The row to look at is the third one. On the traditional clouds the semantic layer is either absent or belongs to the BI tool, which is precisely the failure the framework identifies. That is not a small gap, and it is the clearest thing this framework gets right about the competitive position.

Table of Contents