Databricks Files API 3.0: multipart uploads and conditional writes

An upload finishes, but its response never reaches the client. Should the retry overwrite the path? We'd answer no for an ingestion landing file, because a second writer may have put different output there. Databricks' 23 October 2026 Files API 3.0 Beta gives that decision a server-side check through conditional writes to Unity Catalog volumes.

For TechFabric, the adoption decision would be whether to pause an ambiguous ingestion or let it repair itself by overwriting. We'd choose the pause. That means accepting manual investigation when reconciliation lacks evidence, even if an overwrite would get the schedule moving again. We haven't validated this Beta with our ingestion clients.

Decide what an existing file means #

The conditional-request documentation says If-None-Match: * creates a file only when the path doesn't exist. An existing file produces 412. That protects the earlier content, but it doesn't tell the retry whether that content is its own.

For the lost-response example, our caller would need these distinct outcomes:

A 412 is not a successful ingestion.

For a small executable worked example, this Python helper models the decision after a conditional create. same_output means the application has independently verified the intended source and content. It isn't inferred from the status code.

def next_step(status, same_output=None):
    if status == 200:
        return "continue"
    if status == 412 and same_output is True:
        return "reconcile"
    return "stop"

assert next_step(200) == "continue"
assert next_step(412, True) == "reconcile"
assert next_step(412, False) == "stop"
assert next_step(412) == "stop"
assert next_step(None) == "stop"  # Lost response, outcome unknown.

The missing-information case deserves an assertion of its own. Otherwise, "no mismatch found" can quietly become "same output verified" and let an unreadable file through.

We'd spend the implementation effort on retaining the source identity and recording what comparison was possible, so an operator arriving after a failed retry can distinguish an earlier successful upload from a different writer's output without guessing which file to keep. "File exists" doesn't answer that question.

Keep replacement outside this helper. A person choosing to replace a conflicting file should make a new, explicit decision after inspection, rather than inherit an overwrite concealed inside retry logic. That separation makes recovery slower when nobody is available. We'd take that cost over silently removing another writer's evidence.

A long upload can lose the race at completion #

For multipart uploads, the conditional-request reference puts the condition on initiation and evaluates it at completion. Start one upload, let a competing writer create the target, then complete the first upload. The acceptance test should expect a conflict without replacing the competing file.

Don't start downstream processing just because all the parts were sent. The multipart protocol has a separate completion step; its response belongs in the recovery logic too.

For the lost-response case, deliberately suppress the response in a test client after a successful upload, then restart the caller with the same source identity and verify that it recovers without overwriting the existing file or starting downstream work twice. Run the competing-writer case separately with different content, so a convenient identical file can't conceal the bug.

The helper proves a caller decision, not Files API behavior or exactly-once downstream processing. Those need the development trial, including a durable record of downstream work so restarting the client can't start it twice.

Check the version differences and network requirements before running it under the intended Unity Catalog grants. Keep production writers on 2.0 until both failures are recoverable. We'd rather stop an ambiguous ingestion than erase the evidence needed to repair it.