Coordinator/worker topology

The coordinator/worker topology enables deploying orchestrator instances on standalone machines with zero infrastructure dependencies — no PostgreSQL, no S3 credentials. Workers are the same orchestrator binary running in a different mode, connected to a coordinator via P2P WebSocket.

Design principles

Same binary, different mode. Workers and coordinators are the same @kici-dev/orchestrator package. KICI_CLUSTER_ROLE=worker|coordinator (default: coordinator) controls which subsystems initialize. No separate package, no separate binary.
Workers are stateless. All persistent state lives on the coordinator (PG, secrets, config). Workers use in-memory stores and relay results back.
Workers need only WebSocket connectivity. A worker needs to reach the coordinator’s /ws/peer endpoint. No database, no Platform connection, no S3 credentials required.

Subsystem skip table

When an orchestrator starts in worker mode, the following subsystems are skipped:

Subsystem	Coordinator	Worker	Notes
PostgreSQL connection	Yes	Skipped	Workers have no `KICI_DATABASE_URL`
Outbound WebSocket to KiCI	Yes	Skipped	Only the coordinator connects upstream; workers reach the rest of the cluster through it
Raft consensus	Yes	Skipped	No leader election on workers
PG-backed job queue	Yes	Skipped	Workers NAK if they can’t dispatch immediately
PG-backed execution tracker	Yes	Skipped	Workers use `InMemoryExecutionTracker`
PG-backed agent token store	Yes	Skipped	Workers use `StaticAgentTokenStore`
Webhook pipeline (dedup, trigger matching)	Yes	Skipped	Coordinator handles all webhook processing
Secrets store (PG/Vault)	Yes	Skipped	Workers receive secrets via job reroute payload
Config store (PG SharedConfig)	Yes	Skipped	Workers use local YAML/env config only
Workflow registration store	Yes	Skipped	Coordinator manages registrations
Agent registry	Yes	Yes	Workers manage their own local agents
Scaler manager	Yes	Yes	Workers spawn agents from local `scalers.yaml`
Dispatcher	Yes	Yes	Workers dispatch to local agents
P2P peer client	Optional	Yes	Workers connect to coordinator via P2P
P2P peer handler	Yes	Optional	Coordinator accepts incoming peer connections
HTTP health endpoints	Yes	Yes	`/health`, `/ready`, `/cluster/health`
Worker status endpoint	No	Yes	`/status` with recent job history
Worker drain endpoint	No	Yes	`POST /drain` for graceful drain

Plan limit

Every orchestrator instance counts toward the organization’s plan orchestrator limit — coordinators and workers alike. A worker does not open its own upstream connection, so the coordinator reports its connected workers to the hosted Platform. The Platform aggregates the org’s total (one coordinator plus its workers, summed across every coordinator) and pushes each coordinator a ceiling on how many workers it may hold. The coordinator refuses a worker join past that ceiling. An organization that stays over its limit is notified, and after a grace window its newest workers are drained and disconnected down to the limit.

Job flow

GitHub/webhook --> Platform relay --> Coordinator orchestrator
                                       |
                                       v
                                 Trigger matching
                                 (lock file eval)
                                       |
                          +------------+------------+
                          |                         |
                    Local agents?              Peer workers?
                          |                         |
                    Dispatch locally          job.reroute
                                                    |
                                        +-----------+-----------+
                                        |                       |
                                   Worker A                Worker B
                                   (accept)                (accept)
                                        |                       |
                                  Agent dispatch          Agent dispatch
                                        |                       |
                                  job.progress             job.progress
                                  (relay back)             (relay back)
                                        |                       |
                                        +-----------+-----------+
                                                    |
                                              Coordinator
                                         (check run update)

NAK-based flow control

A worker holds an in-memory queue of the jobs it accepted. When a worker receives job.reroute:

If it can dispatch the job to a matching agent, or queue it for an agent its scaler spawns —> ACK
If it cannot handle the job (draining, dispatch rejected) —> NAK with reason

The coordinator tracks NAK count per peer with backoff. After repeated NAKs from the same peer, the coordinator deprioritizes that peer temporarily.

If a worker ACKs a job but the scaler then fails to provision an agent (Docker down, Firecracker exhausted), the coordinator re-dispatches the job (to another peer or locally) rather than stranding the run. A bounded post-ACK spawn window backstops this: the first progress from the accepting peer disarms it, and a worker’s report that its spawn retries ran out re-dispatches the job immediately (see Worker spawn retries). A worker relays progress because it holds no database. A peer coordinator instead writes the job’s status to the shared database and relays nothing. So when the window elapses, the coordinator reads that row and disarms the backstop for a job the peer already started. The re-dispatch, the disarm, and a job’s terminal cleanup are all keyed to the peer the job is tracked against. So the coordinator ignores a stale signal relayed for the same job by a superseded peer, after an earlier re-dispatch moved it. Such a signal cannot bounce the healthy replacement, wrongly mark the run finished, or strip the replacement’s spawn-window backstop.

Worker spawn retries

A worker retries a failed agent spawn for a rerouted job, within a budget the coordinator sends on job.reroute (spawnRetry: { maxAttempts, backoffMs }, resolved from the org’s reroute_spawn_max_attempts / reroute_spawn_retry_backoff_ms).

One spawn at a time. Every scaler request for the job passes a per-job attempt gate. While a spawn is in flight, or during the backoff after a failure, the gate skips the job, so a capacity-freed re-drive cannot start a second spawn. When the backoff ends, the worker offers the job to its scaler again. The gate records which agent the in-flight spawn creates, and only that spawn’s failure counts against the budget. A failure from an earlier spawn whose job was handed to another agent and then requeued is ignored, so it cannot open the gate while a newer spawn is in flight.
A verdict on every failure. The worker relays each scaler.failed for the job on scaler.event with final: false while attempts remain and final: true on the last one. A failure that arrives after an agent received the job is not a spawn failure, and its relay carries no verdict. A second report of the same failed spawn also carries no verdict, so the coordinator counts each spawn once.
Exhaustion gives the job back. After the final relay, the worker removes the job from its queue. It reports no terminal status for the job: the coordinator owns that verdict and re-dispatches or fails the job.
A cancel removes a queued job. On peer.job.cancel the worker removes the matching queued jobs and reports them dequeued. A dispatched job still gets its job.cancel, and its queue entry is stop-marked, so a requeue after its agent drops it does not bring it back.

Reliable terminal-status relay

A worker holds no database, so a job’s terminal status (success, failed, cancelled, …) exists only in the worker’s memory until the coordinator records it. To survive a connection drop between “the job finished on the worker” and “the coordinator wrote the result”, the worker keeps a durable on-disk outbox of terminal job statuses.

The flow for a job-level terminal job.progress:

Persist before send. When a rerouted job reaches a terminal state, the worker writes the terminal job.progress to its outbox (an fsync’d file under the worker’s data directory) before sending it over the peer connection. The record is keyed by (coordinator URL, runId, jobId).
Send live. The worker sends the job.progress to the owning coordinator.
Coordinator acknowledges. Once the coordinator has applied the terminal status to its run/job rows, it replies with a job.progress.ack carrying the same (runId, jobId).
Prune on ACK. On receiving the ACK, the worker removes the matching outbox record. A record that is never acknowledged stays on disk.
Replay on reconnect. Every time the worker (re)connects to a coordinator, it replays all outbox records destined for that coordinator. The coordinator applies each terminal status idempotently ((runId, jobId) is the dedup key) and acknowledges it, so a status is delivered at least once over the wire and applied exactly once.

Records that are never acknowledged — for example because the coordinator was permanently replaced — are pruned after a 24-hour retention window so the outbox cannot grow without bound. Only terminal job-level statuses are relayed this way; step-level progress and log chunks remain fire-and-forget real-time relay.

Rerouted-job recovery guard

The coordinator’s execution_jobs table marks every job dispatched to a remote worker with a rerouted_to_peer column (the worker peer’s instance id; NULL for locally dispatched jobs). Run-recovery — both orphan recovery and stale-run detection — reads this marker so it does not race the durable relay above:

While the owning worker peer is connected, a rerouted job’s in-flight status is left alone. Its terminal status may still arrive (or be replayed from the worker’s outbox), so failing it early would discard a result that is on its way.
Once the owning worker peer disconnects, the deferral ends and the rerouted job is failed like any other orphan, so a job on a dead worker can never hang indefinitely.

Locally dispatched jobs (rerouted_to_peer = NULL) are never deferred — recovery handles them immediately.

Log relay

Workers relay log chunks to the coordinator in real-time:

Agent --> Worker (log.chunk) --> Coordinator (peer.log.chunk) --> log storage
                                                              \-> Platform --> Dashboard

A worker owns no log storage. The coordinator that dispatched the job persists every relayed chunk under executions/{runId}/job-{name}/step-{index}.log, keyed by the same job name that fills the step’s log_path, and forwards it to the Platform for browser fan-out — so a worker-dispatched job’s logs are stored and viewable exactly like a locally-dispatched one’s, with the originating stream (stdout / stderr) preserved per line. The coordinator also feeds a short in-memory tail used by check-run summaries.

Log chunk relay is fire-and-forget (no ACK required). Some log loss is acceptable for real-time relay.

Cache relay

Workers relay cache operations through the coordinator:

Cache download (agent needs source/deps)

Agents receive pre-signed S3 GET URLs in the job.reroute payload (sourceTarUrl, depsUrl). Direct S3 download, no credentials needed on the worker. Agents extract the raw .kici/ directory from the tarball and import the workflow .ts via the shared TypeScript loader hook — no pre-bundled artifact.

Cache upload (build job results)

Agent --> Worker (cache.upload.request) --> Coordinator (generate pre-signed PUT URL)
                                        <-- Worker (cache.upload.response with URL)
Agent --> S3 (direct upload via pre-signed URL)

The coordinator holds S3 credentials and generates pre-signed URLs. Workers and agents only need HTTP access to the S3 endpoint.

Version compatibility

Version compatibility uses a protocol version integer as the baseline gate. Capability schemas carry per-connection data above it.

Protocol version (baseline gate)

Defined in packages/engine/src/protocol/version.ts as PROTOCOL_VERSION and MIN_PROTOCOL_VERSION. All three connection types enforce minimum-version semantics:

Platform-orchestrator: Platform handler rejects connections with protocolVersion < MIN_PROTOCOL_VERSION (WS_CLOSE_PROTOCOL_ERROR)
Coordinator-worker: Peer handler rejects connections with protocolVersion < MIN_PROTOCOL_VERSION (WS_CLOSE_PROTOCOL_ERROR)
Agent-orchestrator: Agent handler rejects connections with protocolVersion < MIN_PROTOCOL_VERSION (WS_CLOSE_PROTOCOL_ERROR)

Future protocol versions are always accepted (minimum-version semantics, not exact-match). The protocol version is incremented when a message schema gains something an older peer cannot parse. PROTOCOL_VERSION and MIN_PROTOCOL_VERSION are both 4: a peer on protocol 3 (an orchestrator, agent or peer from 0.15.x or earlier) or older is refused at connect with WS_CLOSE_PROTOCOL_ERROR and the reason Unsupported protocol version. Version 4 exists because it requires message fields that a version-3 peer may omit.

The minimum accepted version is pinned to the current version, so raising PROTOCOL_VERSION raises the floor: every node must be upgraded in the same window, and no down-convert path for an older peer exists.

Capability schemas

Peer capabilities (peerCapabilitiesSchema): exchanged in peer.auth.response and peer.heartbeat. Current flags: s3LogAccess, logRoutingOverride.
Orchestrator capabilities (orchCapabilitiesSchema): sent in auth.request. They carry the orchestrator’s role and its dashboard settings: orchRole, dashboardWrites, dashboardEncryptionKey and dashboardVerifiedIssuer.
Platform capabilities (platformCapabilitiesSchema): advertised once after authentication as a standalone platform.capabilities message. Protocol 4 defines no flag.
Orchestrator to agent capabilities (orchAgentCapabilitiesSchema): advertised on register.ack. Protocol 4 defines no flag.
Agent to orchestrator capabilities (agentCapabilitiesSchema): advertised on agent.register. Protocol 4 defines no flag.

Schemas use .passthrough() so newer peers sending unknown flags don’t get stripped. Missing flags default to false (unsupported).

Software version (diagnostic only)

Both sides send softwareVersion in the peer auth handshake for logging and debugging. It is not used for compatibility gating. peer.hello lists the authentication schemes the accepting side supports (authSchemes), and both sides require mutual-v2. Every coordinator and worker in a cluster runs a release with mutual peer authentication, so upgrade them together: an orchestrator with it and an older one do not connect to each other.

Capability manifest (REST)

A REST client does not receive the WS-layer capability handshake. The orchestrator exposes a public GET /api/v1/capabilities endpoint that returns the same three version fields (orchestratorVersion, protocolVersion, minProtocolVersion). The endpoint is unauthenticated, matching the security posture of /health, so a client can read the versions even when its authenticated calls fail.

Stale eviction

The coordinator auto-evicts stale workers after 2 missed heartbeats (default 60s at 30s interval). Configurable via KICI_CLUSTER_PEER_STALE_TIMEOUT_MS.

On eviction:

Worker marked as disconnected in peer registry
No further jobs routed to the evicted worker
In-flight rerouted jobs on the evicted worker are failed by run-recovery once the worker disconnects (see rerouted-job recovery guard); while the worker is still connected, recovery defers to the durable terminal-status relay

Worker identity

Workers use the same KICI_CLUSTER_INSTANCE_ID config as coordinators. Default: random UUID. Operators can set a human-readable ID (e.g., mac-mini-1).

All log lines include the orchestrator’s identity in structured fields (kici.instanceId, kici.role). For worker-executed jobs, the worker’s identity traces which machine ran a job in ELK.

Dashboard visibility

Workers appear in the same peer list as coordinators in the diagnostics dashboard, with a role badge distinguishing “coordinator” (green) and “worker” (blue). No separate UI section for workers.