Start here · Chapter 01
Welcome
This is the manual we wish someone had handed us the first time we stood up a Databricks platform for a client, and it is the manual our own engineers work from today.
The short version
This is a free, opinionated manual for the Databricks platform, written by the engineers at TechFabric who build on it for a living. It explains what the platform is, what it replaces, what it costs, and how to run it in production, in plain language first and in depth second. Every chapter opens with a summary like this one, so a decision maker can read the summaries alone in about twenty minutes and come away with an accurate picture, while an engineer can go as deep as the work requires. It is honest about the places a simpler tool or another cloud is the better answer, because a recommendation you cannot trust to say no is not a recommendation.
TechFabric has been building software since 2017, and a large part of that work now lands on Databricks. We have run migrations off legacy warehouses, built governed agent platforms, and taken more than a few proofs of concept from a notebook that worked on a Tuesday to something a business could depend on. Along the way we accumulated opinions, some of them the obvious ones every good team converges on and some of them the result of getting it wrong first. This manual is where we write them down.
Two readers, one manual #
Almost every chapter has to serve two people who want different things, so rather than writing two manuals we marked the difference on the page.
Every chapter opens with a block titled The short version, written for someone who needs to understand what a thing is and why it matters without learning how it works. If you are deciding whether to fund something, reading only those blocks from front to back takes about twenty minutes and will give you an accurate picture.
Everything after that block is written for the person who has to build it. It goes basics first and then into the capabilities most people do not know exist, because a fair amount of the resistance we meet comes from comparing a platform against a half-remembered feature list.
The other three kinds of block #
Three more structures appear throughout, and they mean specific things.
A side by side block compares Databricks against the named services you would use on AWS, Azure or GCP to do the same job. We argue for Databricks in these, because we have a view, and we name the places the other clouds genuinely win rather than pretending they do not exist. Databricks Versus the Traditional Clouds collects the whole comparison in one place if that is what you came for.
A what people say block states an objection in the words people actually use, then answers it. These exist because the same handful of objections come up in every architecture review, and a manual that ignored them would be less useful than one that takes them seriously. If you are reading this because a colleague sent it to you to change your mind, those blocks are the honest part.
Callouts marked Our position mark places where reasonable teams disagree and we have picked a side. They are the paragraphs most worth arguing with, and they tell you more about how we work than any capability list would. Callouts marked Warning describe something that has cost us or a client real time, money, or trust.
How to read it #
The parts follow what a platform actually does: storing, processing, analysing and serving, then operating it once it exists. Reading straight through builds a platform from the account downward, and almost nobody reads a manual that way.
If Databricks is new to you, start at The Vocabulary before anything else, then do the hour of hands-on building in Getting Started Free. Those two chapters define every term the rest of the manual leans on, including what a notebook, a cluster and Spark actually are, and they get you from nothing to a working governed table without spending anything.
If you are evaluating whether to move at all, read What Databricks Actually Is, then Why Not Just Postgres and a Dashboard?, then What It Actually Costs. Those three are written to be read in order and to be honest about when the answer is that you should stay where you are.
If you have already decided and need to build it, start at Accounts and Workspaces and Unity Catalog, because those two decisions are the ones that are cheap now and expensive to change later.
If you are here for reporting specifically, the three chapters that matter are Reporting, Semantics, and Power BI, Multi-Tenant and Per-User Reporting for showing each customer only their own data, and The Five Pillars of Modern Analytics for the framing underneath both.
If somebody has told you a specific thing is impossible or pointless, the chapter on that thing probably addresses it directly. Lakebase exists in its current form because "it is just Postgres" is the most common thing we hear and the most wrong, and LTAP exists because the thing that replaces it is new enough that most people have not met it yet.
Press / anywhere to search. Results point at the specific section that matched rather than the top of a chapter.
What this manual is not #
It is not a replacement for the Databricks documentation, which is good, and which we link to rather than paraphrase. It is not a certification study guide. It is not exhaustive, because there are corners of the platform we have not needed in production and we would rather say nothing than say something confident and wrong.
It is also not finished. Databricks ships quickly and renames things while it does, so anything written here has a shelf life, and this manual is maintained rather than published once. Where a feature is new enough that our opinion is provisional, we say so instead of projecting more certainty than we have.
A note on names #
Databricks product names have moved around, and the manual uses whatever the platform calls a thing at the time of writing while mentioning the older name once so that search still finds it. If a name here has drifted from what you see in the console, the concept underneath is almost always the same. The Glossary carries both names where they differ.