# Databricks transformWithState async processing: when we'd switch

> Async processing for Python transformWithState is GA on Databricks Runtime 19. When we would convert a stateful processor, and what to check first.

By Ihor Seleznov, 23 October 2026. Databricks Field Guide, TechFabric. Canonical: https://databricks.techfabric.com/notes/asynchronous-processing-with-transformwithstate-is-now-generally-available/

We'd split an asynchronous processor migration into two changes: first make the code safe to interleave, then change the processor class. Databricks made Python's row-based `transformWithState` asynchronous processing [generally available on 19 October 2026](https://docs.databricks.com/aws/en/release-notes/product/2026/october#asynchronous-processing-with-transformwithstate-is-now-generally-available). The temptation is to make both changes together and judge the result by how fast the stream runs.

That would make a wrong answer harder to diagnose. We'd accept the extra review and release rather than mix a correctness fix with a scheduling change. A processor already keeping up with its trigger interval can wait. We haven't benchmarked this release on our streams.

## Make the interleaving visible

The [async reference](https://docs.databricks.com/aws/en/stateful-applications/async) says handlers can interleave across grouping keys and warns about member variables and external systems. That makes a field on `self` worth inspecting before changing `def` to `async def`.

This runnable Python example isolates the risk. It uses ordinary `asyncio` to model interleaving, without calling Databricks. The event deliberately holds the first call while the second call changes the shared field.

```python
import asyncio

class SharedKey:
    async def handle(self, key, release):
        self.key = key
        await release.wait()
        return self.key

class LocalKey:
    async def handle(self, key, release):
        await release.wait()
        return key

async def trial(processor):
    release = asyncio.Event()
    first = asyncio.create_task(processor.handle("first", release))
    await asyncio.sleep(0)  # Let the first call reach its wait.
    second = asyncio.create_task(processor.handle("second", release))
    await asyncio.sleep(0)  # Let the second call reach its wait.
    release.set()
    return await asyncio.gather(first, second)

async def main():
    assert await trial(SharedKey()) == ["second", "second"]
    assert await trial(LocalKey()) == ["first", "second"]

asyncio.run(main())
```

The first assertion passes because both calls return `"second"`. No exception tells the caller that the first answer belongs to the wrong key.

The local variable fixes this example. It doesn't establish that the rest of a processor is safe.

For the first review, list the shared objects and write down what must remain true when two handlers overlap, then put that forced ordering in a regression test so a later refactor can't quietly restore the field that broke the association. A counter, a cache and a client connection may need different treatment. Keep that review small enough to reason about without a throughput graph.

```mermaid How we would decide whether to keep an asynchronous processor
flowchart TD
  A["Review shared objects and repeatable external calls"] --> B["Trial one async processor on its checkpoint"]
  B --> C["Compare batch duration and external service behavior"]
  C --> D{"Useful improvement without a correctness regression"}
  D -->|"Yes"| E["Keep the async class"]
  D -->|"No"| F["Restore the synchronous class"]
```

## A reversible class change isn't a reversible side effect

The [reference](https://docs.databricks.com/aws/en/stateful-applications/async#run-a-query-with-an-async-processor) allows checkpoint reuse between synchronous and asynchronous processors. Keep that route back for the second change, but don't confuse it with reversing a request already sent to another service.

Our adoption rule would exclude a processor with an unexplained external side effect. It could enter the trial after its owner supplies a safe repeat strategy; until then, a backlog is preferable to an untraceable duplicate write.

With those conditions met, trial the class change in a development job and compare batch duration against the trigger interval, alongside the external service's load. Our [streaming chapter](15-real-time-and-cdc.md#operating-stateful-streams) covers those operating measures. The Python assertions establish a local race and its local fix; Databricks compatibility and performance remain untested.