Start here · Chapter 06

Getting Started Free

Almost everything this manual describes can be practised on a permanently free account, and we think you should stop reading at the end of this chapter and go and do it. What follows is a guided path with real code, and each step names the chapter that explains the idea properly once you have seen it work.

What you actually get #

Free Edition is a single workspace with a single metastore, attached to an account that has no account console and no account-level API. Compute is serverless only, with a limited size and limited usage and no custom compute configurations to tune. You get one SQL warehouse fixed at 2X-Small, a maximum of five concurrent job tasks, one active pipeline per pipeline type, and up to three Databricks Apps, which stop themselves after twenty-four hours.

The generative side is present but bounded, with one AI Search vector search endpoint limited to one search unit, one Lakebase project with scale-to-zero compute, and model serving with endpoint limits but no GPU serving, no provisioned throughput and no batch inference. Verifying your account with LinkedIn unlocks limited serverless GPU access and outbound internet connectivity. Do this before you start rather than when something fails, because any step that installs a package or reaches an external service quietly depends on it.

Unity Catalog, notebooks, Genie, AI/BI dashboards, jobs and the Databricks Academy self-paced training are all included, and those are what this chapter needs.

Some things are absent rather than limited. R and Scala are unsupported, so everything here is Python and SQL, and custom workspace storage locations, online tables, clean rooms, Knowledge Assistant and the legacy features are unavailable. Authentication is email one-time password, Sign in with Google, or Sign in with Microsoft, with no SSO and no SCIM, and there are no compliance controls, no private networking, no SLA and no support. If you exceed a quota the compute shuts down rather than billing you, and accounts left inactive for a long time may be deleted, so keep nothing here you would be sorry to lose.

Step 1: Create the account and find your way around #

Sign up at databricks.com/learn/free-edition with an email address, a Google account or a Microsoft account. There is no card and no trial clock. When the workspace opens, spend five minutes on the left-hand navigation rather than diving in, because the shape of that menu is the shape of the platform. Catalog is Unity Catalog and therefore governance, SQL Editor and Dashboards are the analytical surface, Workspace holds your notebooks, Jobs and Pipelines are orchestration, and Compute offers a serverless option with nothing to configure.

Before any code will run, you need somewhere to run it, and on Free Edition that is deliberately simple. For SQL, open the SQL Editor from the left navigation and it is already connected to the one serverless warehouse the account includes. For Python, choose New and then Notebook from the top of the navigation, and in the compute selector at the top right of the notebook pick Serverless. Every SQL block in this chapter runs in either place, and every Python block runs in a notebook cell. The filenames on the code blocks are the names we suggest you give each notebook as you go, so that at the end you have a small ordered project rather than one long scroll.

Create a catalog and a schema to work in, so that nothing you build lands in a default location you will later have to clean up, and a volume, which is the governed folder your raw files will land in before they become tables.

CREATE CATALOG IF NOT EXISTS school;
CREATE SCHEMA IF NOT EXISTS school.demo;
CREATE VOLUME IF NOT EXISTS school.demo.landing;

Step 2: Load data and make a governed table #

Find a CSV you care about, ideally from your own world rather than a sample dataset, because the questions you ask later will be better. Upload it into the volume you just created: open Catalog in the left navigation, click through school, then demo, then landing, and use the Upload button to drop the file in. Then load it yourself rather than letting the interface create the table for you, so that you see the mechanics. If your file has different columns, change the table definition to match; the point is the shape of the flow, not our column names.

CREATE TABLE IF NOT EXISTS school.demo.orders_raw (
  order_id     STRING,
  customer_id  STRING,
  order_date   DATE,
  total        DECIMAL(10, 2),
  status       STRING
);

COPY INTO school.demo.orders_raw
FROM (
  SELECT
    order_id,
    customer_id,
    CAST(order_date AS DATE)        AS order_date,
    CAST(total AS DECIMAL(10, 2))   AS total,
    status
  FROM '/Volumes/school/demo/landing/'
)
FILEFORMAT = CSV
FORMAT_OPTIONS ('header' = 'true');

COPY INTO is idempotent, meaning it remembers which files it has already ingested, so running it twice does not double your rows, and that single property is the difference between a load you can rerun safely and one you cannot. Ingestion explains where it stops being the right tool.

What you have made is not a file, it is a table with an owner, grants, a schema and a history. Tables and Storage covers what Delta is doing underneath, and Unity Catalog covers why the three-level name matters more than it looks.

DESCRIBE HISTORY school.demo.orders_raw;
GRANT SELECT ON TABLE school.demo.orders_raw TO `account users`;

Step 3: Build the medallion layers on it #

Now do the thing every real project does, which is to separate what arrived from what you trust. Bronze is the raw landing you already have. Silver is cleaned, typed and deduplicated. Gold is the shape a dashboard or a model actually wants.

CREATE OR REPLACE TABLE school.demo.orders_silver AS
SELECT
  order_id,
  customer_id,
  order_date,
  total,
  lower(trim(status)) AS status
FROM school.demo.orders_raw
WHERE order_id IS NOT NULL
  AND total >= 0;

CREATE OR REPLACE TABLE school.demo.orders_gold AS
SELECT
  date_trunc('MONTH', order_date) AS order_month,
  status,
  count(*)                        AS order_count,
  sum(total)                      AS revenue
FROM school.demo.orders_silver
GROUP BY 1, 2;

Three statements is a toy version of a pattern that carries very large estates, and Medallion Architecture explains why it carries them. Notice that you have not yet needed a pipeline, a scheduler or a cluster configuration.

Step 4: Query it, build a dashboard, then ask Genie #

Open SQL Editor, point it at your one 2X-Small warehouse, query the gold table, then create an AI/BI dashboard over the same query and publish it. That takes about ten minutes and gives you what most people are actually asking for when they ask for a data platform.

Then create a Genie space scoped to your schema and ask it a question in English. It will be roughly as good as the descriptions you have written, which is the lesson. Add a comment to a column, ask again, and watch the answer improve.

COMMENT ON TABLE school.demo.orders_gold IS
  'Monthly order counts and revenue by order status, one row per month and status.';

ALTER TABLE school.demo.orders_gold
  ALTER COLUMN revenue COMMENT 'Sum of order totals in GBP, excluding cancelled orders.';

Consumption covers the query side and Reporting covers dashboards, metric views and what makes a Genie space trustworthy rather than merely impressive.

Step 5: Stand up a Lakebase project #

Free Edition includes one Lakebase project with scale-to-zero compute, so you can see the transactional half of the platform without paying for an idle database. Create the project from the workspace navigation, then connect with any Postgres client or with psql, because it is Postgres and it behaves like Postgres. The block below is Postgres SQL and runs against that connection, not in the Databricks SQL Editor, which is why its filename does not say notebooks.

CREATE TABLE app_sessions (
  session_id   TEXT PRIMARY KEY,
  customer_id  TEXT NOT NULL,
  started_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_seen_at TIMESTAMPTZ
);

INSERT INTO app_sessions (session_id, customer_id)
VALUES ('s-1001', 'c-42');

The interesting part is not that Postgres works, it is that this database sits inside the same governance boundary as the tables from step two. Lakebase explains the separation of storage from compute that makes scale-to-zero possible, and LTAP explains why both shapes of work over one copy is the argument rather than a convenience.

Step 6: Call a foundation model from SQL and from Python #

This is the step that surprises people, because a language model call is a SQL function, which means classification and extraction can live inside a pipeline rather than beside it.

SELECT
  order_id,
  status,
  ai_query(
    'databricks-meta-llama-3-3-70b-instruct',
    'Classify this order status as one of GOOD, RISKY or UNKNOWN. ' ||
    'Reply with one word only. Status: ' || status
  ) AS risk_label
FROM school.demo.orders_silver
LIMIT 20;

Keep the LIMIT while experimenting, because every row is a model call and Free Edition's endpoint limits will stop your compute rather than send you a bill. The same endpoints are reachable from Python through an OpenAI-compatible client, which is the shape most application code uses.

# Cell 1: the Databricks SDK is preinstalled and already authenticated
# in a notebook, so it is the reliable way to reach the endpoint.
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
client = w.serving_endpoints.get_open_ai_client()

response = client.chat.completions.create(
    model="databricks-meta-llama-3-3-70b-instruct",
    messages=[{"role": "user", "content": "Summarise what a medallion architecture is, in two sentences."}],
)
print(response.choices[0].message.content)

If that model name is not available on your account, open Serving in the left navigation and use the name of any chat model listed there, since the available endpoints change over time and the code does not care which one it talks to.

Foundation Models and the AI Gateway covers how these calls are governed, logged and rate limited once more than one person is making them.

Step 7: Build a tiny agent over your own table #

An agent is a model call with a tool attached, and the smallest honest version is a function that runs a SQL query and hands the result back as context. Read a few rows from your gold table, put them in the prompt, and ask a question that requires them.

from databricks.sdk import WorkspaceClient

client = WorkspaceClient().serving_endpoints.get_open_ai_client()

rows = spark.table("school.demo.orders_gold").limit(50).toPandas()
context = rows.to_csv(index=False)

answer = client.chat.completions.create(
    model="databricks-meta-llama-3-3-70b-instruct",
    messages=[
        {"role": "system", "content": "Answer only from the table provided. If it is not there, say so."},
        {"role": "user", "content": f"Table:\n{context}\n\nWhich month had the highest revenue, and by how much?"},
    ],
)
print(answer.choices[0].message.content)

That is not a production agent and it is not meant to be, because it has no retrieval, no evaluation and no identity propagation, and Agents on Databricks is the chapter about the ninety per cent of the work this snippet skips. Free Edition's single AI Search endpoint is enough for the next step, which is indexing a text column properly rather than filling a prompt with CSV.

What Free Edition cannot teach you #

We are enthusiastic about this tier and we want to be equally clear about its blind spot, which is that the limits remove precisely the concerns that dominate real platform work.

There is one workspace and one metastore, so there is no development, staging and production topology, no catalog isolation between environments and no promotion path to practise. Authentication is email or a consumer identity provider, so SSO, SCIM and group-driven entitlements cannot be exercised at all. There are no cluster policies, no custom compute configurations and no tagging, so cost attribution, chargeback and budget alerting have nothing to attach to, and the discipline in Cost and Performance cannot be rehearsed. There are no compliance controls and no private networking, so most of Security and Compliance is off the table.

Free Edition teaches the craft, meaning modelling, transformation, querying, governance of objects and the generative surface. It cannot teach the operations, meaning the topology, the identity plumbing, the cost controls and the evidence an auditor asks for, and those are learned on a paid workspace.

The honest reading of that table is that Google's free BigQuery allowance is the best permanent no-cost analytical engine on offer, and if all you want is to run SQL over a few gigabytes for nothing, use it. What none of the other three give a learner is a governed catalog, a warehouse, a transactional database and a model serving layer inside one account under one permission model.

Where to go next #

If you followed all seven steps you have touched governance, storage, transformation, consumption, the transactional side and the generative side, which is the whole map in miniature. The next move is depth rather than breadth, so pick the step that interested you most and read its chapter properly. If none of them did, read Why Not Just Postgres and a Dashboard? again, because the argument for a platform lands differently once you have built something on one.

Table of Contents