Databricks transformWithState async processing: when we'd switch

We'd split an asynchronous processor migration into two changes: first make the code safe to interleave, then change the processor class. Databricks made Python's row-based transformWithState asynchronous processing generally available on 19 October 2026. The temptation is to make both changes together and judge the result by how fast the stream runs.

That would make a wrong answer harder to diagnose. We'd accept the extra review and release rather than mix a correctness fix with a scheduling change. A processor already keeping up with its trigger interval can wait. We haven't benchmarked this release on our streams.

Make the interleaving visible #

The async reference says handlers can interleave across grouping keys and warns about member variables and external systems. That makes a field on self worth inspecting before changing def to async def.

This runnable Python example isolates the risk. It uses ordinary asyncio to model interleaving, without calling Databricks. The event deliberately holds the first call while the second call changes the shared field.

import asyncio

class SharedKey:
    async def handle(self, key, release):
        self.key = key
        await release.wait()
        return self.key

class LocalKey:
    async def handle(self, key, release):
        await release.wait()
        return key

async def trial(processor):
    release = asyncio.Event()
    first = asyncio.create_task(processor.handle("first", release))
    await asyncio.sleep(0)  # Let the first call reach its wait.
    second = asyncio.create_task(processor.handle("second", release))
    await asyncio.sleep(0)  # Let the second call reach its wait.
    release.set()
    return await asyncio.gather(first, second)

async def main():
    assert await trial(SharedKey()) == ["second", "second"]
    assert await trial(LocalKey()) == ["first", "second"]

asyncio.run(main())

The first assertion passes because both calls return "second". No exception tells the caller that the first answer belongs to the wrong key.

The local variable fixes this example. It doesn't establish that the rest of a processor is safe.

For the first review, list the shared objects and write down what must remain true when two handlers overlap, then put that forced ordering in a regression test so a later refactor can't quietly restore the field that broke the association. A counter, a cache and a client connection may need different treatment. Keep that review small enough to reason about without a throughput graph.

A reversible class change isn't a reversible side effect #

The reference allows checkpoint reuse between synchronous and asynchronous processors. Keep that route back for the second change, but don't confuse it with reversing a request already sent to another service.

Our adoption rule would exclude a processor with an unexplained external side effect. It could enter the trial after its owner supplies a safe repeat strategy; until then, a backlog is preferable to an untraceable duplicate write.

With those conditions met, trial the class change in a development job and compare batch duration against the trigger interval, alongside the external service's load. Our streaming chapter covers those operating measures. The Python assertions establish a local race and its local fix; Databricks compatibility and performance remain untested.