Crash and replay
A workflow can wait for days, and the process running it will not live that long. AutoFlow gets restarted, redeployed and rescheduled. Here is why that does not matter to the workflow.
Everything that left the interpreter was recorded
The script runs inside an interpreter that cannot touch the outside world by itself. Every time it reaches out (an action, a timer, a value from a channel) the request and, later, the answer are recorded in the workflow’s history in PostgreSQL. The history, not the process, is the workflow’s memory.
Nothing half-finished is applied
A run proceeds in rounds. A round ends when the script waits for something, and everything the round produced is committed in one transaction at that moment, or not at all. If AutoFlow dies in the middle of a round, the round did not happen: history is exactly as it was before, and the round is done again.
Another instance picks the workflow up
Each AutoFlow instance owns a set of shards, slices of the workflow keyspace. When an instance dies, the survivors claim its shards, and the work that was in flight on them is queued again. The new owner starts the workflow again.
The script re-runs from line 1
There is no snapshot of the interpreter to restore. Instead, the new owner runs the script from the top against the recorded history. Each time the script gathers a future whose answer is recorded, it gets that answer at once and the action is not run again. When it reaches the first step with no answer yet, it has caught up and continues live. This is a cold replay.
Most of the time no crash happened. The instance that ran the previous round still has the workflow’s interpreter suspended in memory, so when the next answer arrives it loads only the new history and continues in place. This is a warm resume. Both paths end in the same state; the difference is how much work it takes to get there.
flowchart TD
A["Workflow is waiting for step 3"] --> B["AutoFlow instance crashes"]
B --> C["Nothing half-finished:<br/>only completed steps are in history"]
C --> D["Another AutoFlow instance<br/>picks the workflow up"]
D --> E["It re-runs the script from line 1"]
E --> F["Steps 1 and 2 return their<br/>recorded results at once"]
F --> G["Execution continues at step 3"]
G --> H["Workflow finishes normally"]
An action can run twice
An action may have done its work at the destination just before a crash, before its result was recorded. History then holds no result for it, so it runs again on replay. Every action therefore carries a stable idempotency key that is identical on every attempt, and the destination can recognize the repeat by it. AutoFlow guarantees the key; the module, or the system behind it, has to honor it.
sequenceDiagram
participant S as The script line calling create_issue
participant F as AutoFlow
participant H as History
participant M as Module
S->>F: create_issue(...)
F->>H: Record step scheduled, queue it for a worker
Note over S: The workflow sleeps here
F->>M: Run create_issue with an idempotency key
M-->>F: Created issue 42
F->>H: Record step finished with issue 42
H-->>S: gather() returns issue 42
Note over H: A later replay returns 42 from history; the module is not called.
Determinism is the price
Replay only works if the script takes the same decisions in the same order every time it runs. The Starlark dialect AutoFlow uses leaves it no other option: a script cannot do I/O, read the clock, or use randomness. time.now() returns the time recorded in history, advancing as results are consumed, so it is the same on every replay. Every side effect goes through an action and is recorded. What you get is a program that can be run again and again against its history and lands in the same place each time.
Two consequences follow. A workflow runs the script it was started with, from start to finish, even if the script in your repository changes. And replay is not free: a cold replay re-executes the script, so a workflow that computes heavily pays that again on every cold start.
For engineers
Replay keys. Every ExecuteActivity and NewTimer call inside the workflow function increments a per-run counter, the replay key. Before the call does anything, autocore looks up history for a scheduled event at that key. If the event exists and has a completion, the call returns an already-resolved future; if it exists without one, a pending future; if there is no event, this is new work and the schedule is accumulated for the next yield. Signals do not consume a replay key, because they are produced outside the function. Inputs are not part of the match: a call is matched by key, event kind and activity name (a timer by its duration), so an input may carry a freshly minted token or timestamp.
Sequence ids. Every history event carries a 1-based, gapless, monotonically increasing sequence_id within its workflow, derived as max(sequence_id) + 1 under a FOR UPDATE lock on the execution row. The log is its own watermark. When several futures are ready at once, a Selector fires the one whose event has the lowest sequence id, so a replay reproduces the live choice.
Warm versus cold. A yielded goroutine is suspended in a bounded pool. Warm resume: the next workflow task is delivered to that goroutine, which loads only the events newer than its last seen sequence id and continues in place. Cold replay: no suspended goroutine exists (evicted after its TTL, pool full, or the shard moved), so a new goroutine loads the full history in one fenced read and runs the function from the top; completed events fast-path through until the first pending one.
Fencing. Each shard carries a fence counter that is bumped on every ownership change, and every write against a shard verifies the fence inside its transaction. A stale owner that comes back after its shard moved cannot commit against the new fence, so it cannot corrupt history.
Step budget. Every interpreter thread of a workflow (the top level, main, one per loaded module file) has a computation-step budget, DefaultMaxFlowSteps, 100,000,000 steps or around a second of CPU. A warm resume carries the count across rounds and a cold replay reproduces it, so a runaway workflow fails at the same instruction wherever it runs, as Failed with an error naming the position. What stays unbounded is the CPU a cold replay re-spends re-executing history.
Replay mismatch. If the Nth scheduling call in a replay does not match the Nth recorded event (a different activity name, a different timer duration, an awaitable child where a detached one was recorded), autocore panics the workflow with a descriptive mismatch error and it terminates as SystemFailed. The sandbox makes this hard to reach from a workflow definition. The realistic hazard is the platform changing under a running workflow: a Starlark parser upgrade that rejects an expression the workflow already used fails the workflow on its next replay.
What all of this adds up to, and where it stops, is on the next page: What AutoFlow guarantees, and what it does not.