How AutoFlow works
A workflow definition is a Starlark program. Running one durably takes four layers, and each of them has one job.
flowchart LR
Caller["A caller in GitLab"]
subgraph AutoFlow["AutoFlow module"]
AF["Starlark interpreter<br/>runs the workflow"]
AC["autocore<br/>durable execution engine"]
end
DB[("PostgreSQL<br/>the history")]
RD[("Redis<br/>which instances are alive")]
MOD["Modules<br/>GitLab API, policy, your own"]
Caller -->|"start a workflow"| AF
Caller -->|"read the result"| AF
AF --> AC
AC --> DB
AC --> RD
AF -->|"run one action"| MOD
MOD -->|"result"| AF
GitLab
A caller starts a workflow through AutoFlow’s gRPC API. The request carries the workflow definition, the arguments for its main() function, a namespace, and an idempotency key. The response carries a workflow key and a workflow token. The call returns as soon as the workflow is recorded; it does not wait for the workflow to run. Later, the caller reads the status and the result with the same token. Deciding who may start a workflow is the caller’s job, not the API’s.
AutoFlow
AutoFlow is the module that does the work. It owns the Starlark interpreter and the translation of what a script does into engine operations: an action call becomes an activity, sleep() becomes a durable timer, a channel becomes a stream of signals. What enters the module is a gRPC call; what leaves it is one action invocation at a time. AutoFlow is hosted in GitLab Relay (formerly KAS), next to Relay’s other modules: the Kubernetes agent server, the Job Router and Workspaces.
autocore
autocore is a generic durable execution engine: it records every step of a workflow as history in PostgreSQL and replays that history to pick the workflow up again. AutoFlow is one domain built on it, by registering a single workflow (run a workflow definition) and two activities (load the definition’s modules, invoke one action). autocore knows nothing about Starlark, and nothing in it is specific to AutoFlow.
A vocabulary note for the rest of this section: a workflow definition is the Starlark program, a workflow is one durable run of it, an action is a function a module exposes to the script, and history is the ordered record of what a workflow has done. The full list is in the glossary.
PostgreSQL and Redis
PostgreSQL holds everything that must survive: the workflow rows, the history, the task queues, the timers, and values sent in from outside. A central database maps idempotency keys to workflows; one or more workflow databases hold the workflows themselves, and capacity grows by attaching another workflow database. Redis coordinates the AutoFlow instances: it tracks which ones are alive and wakes an instance when new work lands on a shard it owns. A shard is a slice of the workflow keyspace, owned by one instance at a time.
Modules
A module gives a workflow the ability to do something outside itself: call the GitLab API, evaluate a policy, talk to a system you run. A module speaks a gRPC and protobuf protocol with AutoFlow, so it can be written in any language; the modules a workflow can load today are built in. What crosses this boundary is an action call carrying a stable idempotency key, its result, and values a module pushes into one of the workflow’s channels.