Analysing and serving · Chapter 21

Serving Data to Applications

Everything up to this point has been about producing trustworthy data. This chapter is about the moment a product asks for it, one customer at a time, while somebody waits for a page to load.

Two shapes of read, and why one warehouse cannot be both #

An analytical read scans a lot of data to answer one question, and it is measured in seconds. A serving read fetches a handful of rows by key, it happens once per user action, and it is measured in milliseconds. The difference is not one of degree.

A SQL warehouse is built for the first shape. Every statement carries planning and scheduling overhead that is irrelevant against a two second scan and dominant against a two millisecond lookup, and concurrency comes from adding clusters rather than from holding thousands of cheap connections. Pointing a product's request path at a warehouse gives you latency that is acceptable in a demo, variable under load, and expensive at any real request rate.

The serving pattern we use #

Gold tables are computed by pipelines in the lakehouse, and the rows a product needs are synchronised into Lakebase, the Postgres-compatible operational layer that sits inside the same account and the same Unity Catalog governance boundary. The application connects to Lakebase with an ordinary Postgres driver, issues ordinary indexed queries, and knows nothing about the lakehouse at all.

This keeps three properties that matter. The read path is a database connection rather than a query engine, so latency is predictable. The analytical estate remains the system of record, so a corrected business rule is fixed once and flows outward. And the serving layer stays derived and disposable, so it can be rebuilt from gold at any time without anybody negotiating about what would be lost.

Application writes are not forbidden by this rule, they belong in their own tables rather than in the synchronised ones. Preferences, review decisions, and workflow state go into the product's own Lakebase schema and sync back into the lakehouse for analysis. What must not happen is a product writing into a table that a pipeline also maintains.

Online features and model inference #

The same problem appears whenever a model is served behind an endpoint. Scoring a request usually needs features computed in batch, such as a customer's rolling averages or a merchant's risk history, and fetching them cannot involve a Spark job.

Databricks keeps the feature definitions in Unity Catalog and serves the precomputed values from a low-latency store, so an endpoint looks up features by key at request time and one definition covers both training and inference. That prevents training and serving skew, which quietly destroys model accuracy and is almost impossible to detect from outside. The equivalent elsewhere is a feature store product, a separate online store, and the pipeline keeping them in agreement.

Multi-tenant, customer-facing dashboards #

Showing each customer their own numbers is where most teams meet this subject for the first time, and it is worth being precise about where the isolation lives.

For internal or partner users who have identities in your Databricks account, isolation belongs in Unity Catalog. A row filter on a gold table lets one physical table serve every tenant, with each principal seeing only their own rows regardless of which tool they arrive with, so there is no per-tenant copy to keep in step and no filter a report author can forget.

For end customers who will never have a Databricks identity, the enforcement point is the serving layer. We give each tenant a scope enforced by the database rather than by application code, using Postgres row-level security in Lakebase keyed on a tenant identifier taken from the authenticated session, so a missing WHERE clause in one endpoint cannot leak another customer's data. Where regulation or a large customer demands it, the same pattern runs with a schema per tenant at the cost of more objects to manage.

Embedded dashboards are the fast route to a customer-facing analytics feature, and AI/BI dashboards can be published and embedded in an external application. We use that for internal, partner, and lighter customer-facing cases, and we build a bespoke interface served from Lakebase when the requirement is a heavily branded experience, sub-second interactions, or entitlements more complicated than a filter.

Beyond the basics #

Synchronisation modes. A synced table can be refreshed as a snapshot, on a trigger when the source updates, or continuously at low lag. Most product surfaces need far less freshness than their owners claim, so decide the contract per table and publish it, because an unstated freshness contract becomes an argument during an incident.

Branching the serving layer. Because Lakebase separates storage and compute, the read model can be branched. A schema change to a serving table can be tested against a full copy of production data, and a bad rollout is undone by repointing rather than by restoring.

Hosting the application next to the data. Databricks Apps runs a Python or Node web application inside the workspace, with workspace identity, OAuth, and direct access to warehouses and Lakebase. For internal tools, review queues, and the interfaces around governed actions, this removes a deployment target, a separate secret store, and a separate authentication story.

Caching, and being honest about it. The first cache to reach for is precomputation, meaning the aggregate a screen needs exists as a row in gold rather than being computed per request. The second is the serving store itself, which is already a cache with an index. If a workload genuinely needs a hot in-memory tier at high fan-out, run one and invalidate it on pipeline completion rather than on a timer, so freshness follows the data changing instead of a guess.

Getting under 100ms. Keep the request path off the warehouse, precompute and denormalise so a screen is one keyed read rather than six joins, index the serving tables for the access pattern, pool connections in front of Postgres, and keep the application in the store's region. That combination reliably lands in single-digit to low tens of milliseconds. The honest limit is that single-digit milliseconds at extreme fan-out, such as an ad bidder, is still a dedicated key-value store's job.

The last row is the argument. Every service in the AWS column is good at its job, and that is not the problem. The problem is that the reverse ETL job, the cache invalidation logic, the feature synchronisation, and the several permission models are code your team writes, owns, and is on call for, and none of it is the product you are selling.

Table of Contents