Operating · Chapter 29

Migration

Moving an established estate onto Databricks is the engagement we are asked about most often, and it is the one where honest expectation setting does more good than any technical decision.

The three kinds of work #

The three behave differently, carry different risk, and need different people.

Mechanical translation covers table definitions, straightforward SQL, and data movement. It is genuinely well automated, scales with tooling rather than headcount, and is the part vendor demonstrations show. Expect it to go roughly as promised.

Semantic reconstruction covers stored procedures encoding business rules that exist nowhere else, scheduling dependencies implied by a legacy orchestrator's configuration, and definitions living inside a BI tool's semantic layer. None of it is written down, several of the people who wrote it have left, and the only reliable recovery is to read the code and ask questions. This is where migrations overrun.

Consumer migration covers every report, extract, application, and human process reading from the old system. It is not technically hard and it is politically slow, because each consumer has an owner with their own priorities and their own definition of ready.

Sequencing #

We migrate by data domain rather than by layer, and we run the old and new systems in parallel for each domain until reconciliation passes.

Migrating by layer, meaning all bronze then all silver then all gold, sounds orderly and delivers nothing until the last step, so the project carries maximum risk for maximum duration. Migrating by domain delivers a working, verified domain early, which buys the credibility to keep going. The first domain also absorbs the platform setup work described in Accounts and Workspaces and CI/CD, so later domains move faster and an estimate built from domain one errs in a useful direction.

The first domain should matter enough that finishing it means something, be self-contained enough not to drag half the estate behind it, and be owned by someone who wants the migration to succeed. Picking the largest and most contested domain first, on the grounds that it is the hard one, is a common and expensive instinct.

Parallel running is expensive and it is not optional. Reconciliation between old and new is the only evidence that the semantic reconstruction was correct, and it needs enough cycles to catch month-end, quarter-end, and seasonal edge cases. A domain that reconciles for a week has demonstrated less than teams think.

Reconciliation done properly #

Row counts match far more often than values do, and value equality on a sample is weaker evidence than it feels. We compare full aggregate profiles per column, including null rates, distinct counts, and distribution summaries, then compare exactly on a deterministic sample keyed by business identifier, and finally reconcile the handful of numbers the business actually reports.

The last of those is the one that matters politically. When the CFO's number matches, the migration is credible; when it does not, nothing else you demonstrate helps.

Reconciliation should be a scheduled job with stored history rather than a script somebody runs, because a trend of shrinking discrepancies persuades a steering committee better than one passing run, and because the same job becomes your regression test after cutover, turning migration evidence into the monitoring described in Observability.

Fabric Airlift #

Airlift is TechFabric's migration accelerator, and it exists because we kept rebuilding the same artefact bookkeeping on every engagement. It handles inventory and translation across tables, pipelines, and jobs, supports more than thirty source types spanning data warehouses, ERPs, and other systems of record, and turns the mechanical layer into a tracked, repeatable effort that reports what has moved, what has not, and what has been verified.

What it does not do is semantic reconstruction, because that requires judgement about business meaning and we would not trust a tool claiming otherwise. We lead with the distinction because migration tooling is routinely oversold, and a client told the whole thing is automatic will not have staffed for the part that is not.

Advanced: the parts experienced teams plan for and first-timers discover #

Change data capture during the parallel period. The source does not stop while you migrate it. Continuous capture into bronze, rather than repeated full extracts, is what makes parallel running affordable over months instead of weeks, and it turns cutover into stopping writes on the old side rather than a final heroic load. Ingestion covers the mechanics.

Shadow reads for consumers. The lowest-drama cutover we know is to point a report at the new system while still serving the old one, compare the two outputs for a period, and only then flip which one the user sees. The comparison is automatic, the flip is boring, and the rollback is a configuration change.

Behavioural differences that are not bugs. Legacy warehouses differ from Spark SQL in ways that produce different results rather than errors, including implicit rounding, collation and case sensitivity in join keys, null ordering, empty string against null, and timestamps across daylight saving boundaries. Each is a small, plausible discrepancy that costs a day to find. A differences register built on the first domain and applied to the rest stops you finding each of them four times.

Query and workload archaeology. Before translating anything, capture what actually runs against the legacy system, with frequency and consumer. On most estates a substantial share of scheduled jobs feed reports nobody has opened in a year, and the cheapest migration work is the work you delete.

Cost and governance models do not translate. A workload that was economically invisible on a fixed-capacity appliance becomes a visible line item under consumption pricing, and nightly full rebuilds are the usual offender, so rewrite those as incremental before cutover rather than after the first surprising invoice. Legacy grants accumulated over a decade translate no better; copying them faithfully imports a mess you had an opportunity to fix, so we reconstruct entitlements from groups and intent instead.

The row worth dwelling on is federation, because it changes the shape of the project rather than its speed. Querying the legacy warehouse from the new engine lets the reconciliation job read both sides from one place, and lets consumers move to the new endpoint before their data has finished moving. The last row is there because we would rather be the vendor who says it, since no tool on any platform reconstructs undocumented business logic.

Decommissioning #

A migration is not finished when the new system works. It is finished when the old one is switched off, and estates that skip this step pay for both indefinitely while slowly repopulating the old system with new dependencies.

We set a decommissioning date per domain at the start, treat every extension as a decision needing a reason and a named owner, and make the final cutover a scheduled event with a rollback plan rather than a gradual fade. Before switch-off we take a final archival extract into cheap storage with schema and lineage recorded, because the compliance question about a legacy record arrives eighteen months later and the wrong answer is that the system was decommissioned.

The habits a migration forces are worth keeping. Reconciliation, lineage, a written record of what each domain means, and an owner per dataset are things an estate should have anyway, and a migration is the rare moment the organisation will pay for them.

Table of Contents