Multi-orchestrator architecture

KiCI supports running multiple orchestrators in a cluster for high availability, cross-architecture job routing, and dedicated coordinator topologies. This document covers the architecture, protocols, and design decisions.

Design overview

Coordinator/worker model

Every webhook event has exactly one run coordinator — the orchestrator that received the webhook from the Platform relay. The Platform tier uses round-robin load balancing across orchestrators registered with the same routing key; it has no awareness of cluster topology.

The coordinator:

1. Receives the webhook payload from Platform
2. Performs deduplication, trigger matching, and lock file evaluation
3. Creates GitHub check runs as “pending” (immediate user feedback)
4. Claims jobs it can dispatch to its own local agents
5. Reroutes remaining jobs to peer orchestrators with matching agent capacity
6. Tracks job progress from all sources (local agents + peers)
7. Updates GitHub check runs as jobs complete
8. Handles cancel propagation to peers

A peer (worker) that receives a rerouted job:

1. Validates the reroute (loop prevention, capacity check)
2. Dispatches the job to a local agent
3. Reports step-by-step progress back to the coordinator
4. Reports job completion back to the coordinator

Runs that span coordinators

Rerouting is the push path. There is also a pull path: the dispatch_queue table is cluster-wide, and every coordinator claims from it with FOR UPDATE SKIP LOCKED, so an agent connected to any coordinator can claim a job another coordinator queued. Each agent reports to the coordinator it is connected to, and that coordinator writes the job’s status to the shared execution_jobs row. Between coordinators the shared row is the report; no frame is relayed.

A precursor job — a workflow’s __build__, __init__, or dynamic-eval job — produces a result the dispatching coordinator waits for before it dispatches the real jobs. When a sibling coordinator’s agent ran that job, the sibling persists the result to execution_jobs.precursor_result alongside the terminal status. The waiting coordinator’s PendingPrecursorDbWatcher polls the rows it is waiting on and hands each terminal one to settlePendingPrecursor. That is the same function the local agent socket goes through, so the two channels cannot disagree on what “finished” means. A pre-run global eval round is not a run and writes no execution_jobs row, so it has no shared row to read; its own wait ceiling still bounds it. The operator view of the same mechanism is in Clustering → Runs that span coordinators.

Single-orchestrator behavior

When clustering is disabled or no peers are connected, all cluster code is dormant. The coordinator field on the pipeline processor is undefined, and the existing direct dispatch path is used. This ensures zero overhead for single-orchestrator deployments.

Even with clustering enabled, the coordinator path only activates when hasConnectedPeers() returns true. A cluster of one self-elects as Raft leader and operates identically to a non-clustered orchestrator.

Communication topology

Orchestrators participate in three communication channels. Understanding which messages flow where is critical for network planning, firewall rules, and debugging.

Channel summary

Channel	Transport	Messages
Platform ↔ Orchestrator	WS (orch connects outbound to Platform)	Webhook relay, source registration, execution telemetry, peer discovery
Orchestrator ↔ Orchestrator (P2P)	WS (initiator connects to peer’s /ws/peer)	Peer auth, heartbeat (agent inventory + scaler capacity), job reroute, job progress, cancel propagation, log relay, cache upload relay, Raft consensus, operator requests forwarded to one peer or fanned out to connected peers (config reload, log collection, scaler orphans, peer forget, and scaler reload, which also reaches connected workers)
Orchestrator ↔ Agent	WS (agent connects outbound to orch)	Job dispatch, job/step status, log streaming, event emit

What flows P2P vs upstream

The KiCI Platform is primarily a webhook router and peer matchmaker, not a general message bus. Inter-orchestrator coordination never touches the upstream tier: it flows strictly between orchestrators on the /ws/peer channel. It covers peer auth (peer.hello / peer.auth.request), heartbeats (agent inventory, scaler capacity, Raft state, drain status), job rerouting (job.reroute, job.progress, peer.job.cancel, peer.log.chunk), cache upload relay, operator requests (peer.config.reload, peer.logs.collect.request, peer.scaler.orphans.request, peer.scaler.reload.request, peer.forget.request), and Raft consensus. Execution telemetry (lifecycle events, run/job/step status forwards, log chunks) flows the other direction, from each orchestrator up to the Platform tier so the dashboard can render it. See Protocol overview for the message-by-message breakdown across each channel.

Peer communication

Discovery

Peers discover each other through two mechanisms:

1. Platform matchmaker (every Platform-connected mode — platform, hybrid, observed): When an orchestrator sends source.register, the Platform responds with source.register.ack that includes a peers array listing all other orchestrators registered with overlapping routing keys. When an orchestrator registers, the Platform also sends peer.update with the full peer list to every orchestrator already in the pool. Peer matchmaking is independent of relay eligibility — an observed orchestrator never receives a relayed webhook, but it still discovers and is discovered by its peers. An announced address is a hint: a coordinator dials it with the announced instance ID, and the address becomes a peer only after mutual authentication succeeds and the server reports that instance ID. An address that fails mutual authentication is dropped, and dialled again only when the Platform announces it again. A new address announced for a connected peer is dialled as a candidate and replaces the working link only after it authenticates. KICI_CLUSTER_PEER_DISCOVERY=static turns off dialling of announced addresses.

2. Static configuration (independent mode): Operators configure KICI_CLUSTER_PEERS with comma-separated peer addresses. Each orchestrator creates PeerClient instances for all configured peers at startup.

Connection establishment

• Direct WS preferred — orchestrators proactively connect to all known peers
• ECDH encrypted channel — X25519 key exchange derives the handshake key (K_hs), which protects the authentication exchange
• Mutual authentication (mutual-v2) — both sides hash the handshake transcript (both ephemeral keys, the nonce, the advertised schemes). The dialling side proves the shared key with an HMAC over that hash; the accepting side answers with its own HMAC over the hash and the client proof. The shared key is the stored hash of the peer credential (credential mode) or of the join token (token mode). A relay that runs its own key exchange with each side gets two different transcripts, so neither proof verifies across them
• No secret on the wire — in token mode the peer sends the token’s routing part, never the token. The accepting side finds the token row by its routing fields, picks the row the proof matches, and takes the peer’s role and routing key from that row
• Join token or credential — a worker authenticates its first connection with a one-time join token; a coordinator without a token issues its own credential on its first peer connection; every later connection proves the credential
• Application key — after the acceptance both sides switch to K_app, derived from K_hs and the shared key, so a side that skipped the proof check can neither read nor write application frames
• Scheme refusal — a dialling side that does not see mutual-v2 in peer.hello sends nothing secret and closes; an accepting side refuses a request in the earlier scheme with Mutual peer authentication required, which does not count against the rate limit
• Protocol version check — a protocolVersion below MIN_PROTOCOL_VERSION is refused with peer.auth.response { accepted: false } and close code WS_CLOSE_PROTOCOL_ERROR; a newer version is accepted (forward compatibility)
• Auth timeout — both sides close a connection that has not authenticated within 15 seconds
• Rate limiting — 5 failed auth attempts per IP within 60 seconds triggers temporary block
• Auto-reconnect — exponential backoff with jitter (1s base, 1.5x multiplier, 60s max)

Which link carries a message

Two orchestrators can share two links: the one A opened to B, and the one B opened to A. Each link carries messages in both directions once it is authenticated. A Raft broadcast or peer.leaving goes out over every link. A message addressed to one peer (peer.job.cancel, job.reroute) goes over the coordinator’s own outbound link when that link is authenticated, and over the peer’s link to it otherwise. A coordinator’s outbound link is down while it reconnects, and stays down on a coordinator whose credential an operator revoked, which reaches every peer over that peer’s link to it.

• Fire-and-forget messages (peer.job.cancel) try the outbound link first. When that link sends nothing, the message goes over the peer’s link. The outbound link reports a failure only when nothing left the process, so a message is never sent twice.
• A reroute (job.reroute) picks its link before it sends. It is never sent again over the other link: a NAK or an ACK timeout from an authenticated peer is final for that peer, and a second send would deliver the job twice.

Heartbeat

Every 30 seconds (configurable), each peer sends a peer.heartbeat containing:

• Agent inventory: per-agent summary (agent ID, labels, active jobs, max concurrency, platform, architecture)
• Scaler capacity: per-backend summary (label sets, max agents, active count) — enables routing to peers with on-demand scaling capacity even if no agents are currently connected
• Drain status: whether the orchestrator is shutting down gracefully
• Capabilities: feature flags (s3LogAccess, optional logRoutingOverride)
• Raft state: current term and known leader ID
• Config version: shared config version counter for cross-orchestrator config sync detection
• Registry version: workflow registration version counter for cross-orchestrator registration sync
• Timestamp: heartbeat send time (milliseconds)

This inventory is the basis for routing decisions. The coordinator consults the peer registry to find peers with matching capacity — either connected agents with available slots, or scaler backends that can provision agents for the required label sets.

Rerouting protocol

Job reroute flow

The coordinator sends job.reroute over the link the link rule picks.

Reroute message details

The job.reroute message contains:

Field	Description
messageId	Unique message ID for ACK correlation
jobId	Pre-allocated job ID. The sending coordinator allocates it before reroute so its execution_runs / execution_jobs rows reference the same id the receiving peer will dispatch under and forward progress against.
runId	Execution run identifier
deliveryId	Original webhook delivery ID
routingKey	Provider routing key (e.g., github:12345)
event, action	Webhook event type and action
payload	Full webhook payload for independent trigger matching
jobName	Name of the job to execute
workflowName	Workflow containing the job
runsOnLabels	Label sets the job requires
triedConnections	Array of instance IDs already tried (loop prevention)
maxHops	Maximum allowed hops (default: 3)
coordinatorId	Instance ID of the run coordinator
requestId, traceId	Full trace context for distributed tracing
jobConfig	Resolved job configuration (steps, rules, matrix, etc.) so the peer can dispatch without re-resolving
repoUrl, ref, sha	Repository clone URL, git ref, and commit SHA
provider	Provider type (e.g., github)
providerContext	Provider-specific context (e.g., { installationId })
sourceTarUrl, sourceTarDigest	Pre-signed .kici/ source tarball URL + the tarball’s SHA-256 (cache hit)
depsUrl, depsHash	Pre-signed dependency tarball download URL and SHA-256 tarball-bytes hash (cache hit)
runsOnPatterns	Glob/regex include matchers the receiving peer’s agent labels must satisfy, applied on top of the exact runsOnLabels prefilter. A job selected purely by pattern carries its whole selector here.
excludePatterns	Glob/regex matchers that disqualify a candidate agent
excludeLabels	Labels that the dispatched agent must NOT have (optional)
cloneToken	Pre-resolved clone token for workers without provider credentials
encryptedSecrets	Encrypted secrets envelope (AES-256-GCM with session key)
encryptedNamespacedSecrets	Encrypted namespaced secrets envelope

Authoritative source: packages/engine/src/protocol/messages/peer.ts — jobRerouteSchema

Routing algorithm

1. Local-first: The coordinator checks its own agent registry for agents matching the job’s labels
2. Peer fallback: If no local agents match, the coordinator queries the peer registry for peers with matching capacity
3. Capacity sorting: Peers are sorted by available capacity (most capacity first) to balance load
4. Sequential fallback: If a peer rejects or times out (15s), the coordinator tries the next peer
5. Parallel fan-out: Jobs targeting different peers are rerouted concurrently via Promise.all

Local matching and peer-side capacity matching share the same predicate. A label set is considered when it is a superset of the job’s runsOn, none of its labels appear in the job’s excludeLabels, and the job’s runsOn includes every label in that label set’s own opt-in gate. All three tests apply to the same label set. So a scaler mixing platforms is not disqualified by a sibling set’s taint, and a peer is never selected on labels one set supplies while another set’s gate is what was satisfied. The gate is propagated per label set over the heartbeat protocol, so cross-peer reroute decisions enforce it identically.

Exact labels are only the prefilter. A runsOn selector may also carry glob/regex matchers, which travel on the reroute as runsOnPatterns / excludePatterns and are applied as a post-filter on the receiving peer — the same two-stage matching the single-orchestrator dispatch path uses. Carrying them matters most for a job selected purely by pattern: with no exact labels to prefilter on, the patterns are the whole selector.

Loop prevention

Two mechanisms prevent routing loops:

• triedConnections array: Each reroute appends the sender’s instance ID. Recipients reject messages where their own ID is already in the list.
• maxHops counter: Hard limit of 3 hops. Recipients reject messages where triedConnections.length >= maxHops.

Cancel propagation

A run cancel reaches the agent of every job of the run, whichever coordinator receives the cancel. Each message travels over the link the link rule picks.

• Rerouted jobs. When a run is cancelled (via API or fail-fast), the coordinator sends peer.job.cancel to every peer that holds a rerouted job of that run. Cancels are grouped by peer.
• Jobs whose agent is connected to a sibling. The Platform relays run.cancel.request to any connected pool member, so the coordinator that receives it often did not dispatch the job. That coordinator reads, from the host roster, the coordinator each job’s agent is connected to. It then sends peer.job.cancel for the job to that coordinator, with force when the cancel is forced. When that coordinator is alive but no link reaches it, the run stays cancelling and the leader’s stuck-cancelling sweep sends the cancel again later. A forced cancel is recorded on the run (execution_runs.cancel_force), so the sweep sends it again with force.
• The receiver finds the job’s agent in its dispatcher first, and then in the job’s dispatch record (dispatch_queue.agent_id). An agent that reconnected from a sibling, or one that is not back yet after a restart, is known only to the dispatch record. The receiver honours force, and a message with no jobId cancels every job of the run it knows. It logs each cancel it cannot deliver as Peer job cancel not delivered, with the reason.

Post-accept spawn resilience

A positive job.reroute.ack only means the peer queued the job — the agent spawn happens afterwards and can still fail (transient scaler error, image-pull failure, peer crash, OOM). Without a backstop, a spawn that fails after the ACK produces no progress, so the run would sit pending until the stale detector force-fails it minutes later. Two complementary layers make a reroute resilient to a post-accept failure:

• Bounded-window re-dispatch (correctness backstop). When a peer ACKs a reroute, the coordinator arms a per-job spawn window. The first job.progress (job or step) disarms it. If the window elapses with no progress, the coordinator best-effort cancels the original peer’s job (a double-execution guard against a slow-but-healthy spawn), then re-runs the routing decision under the same jobId — trying another peer, then a local dispatch, and finally marking the job failed if no backend can run it. This also covers a peer that dies right after the ACK.

  • Silence is not evidence of failure. Only a worker relays job.progress; a peer coordinator shares the database and writes the job’s status straight to execution_jobs, so it relays nothing at all. Before it treats an elapsed window as a spawn failure, the coordinator therefore reads that shared row. A job the peer has already started disarms the backstop instead of being cancelled, which is what keeps a coordinator-to-coordinator reroute alive past the window (a slow scaler spawn, a long checkout, a big dependency install). The coordinator keeps the job’s tracking entry — cancel propagation still needs it — and polls the row until the job is terminal, which is the only report the peer coordinator ever makes. A read failure is treated as “not started”, so a database fault cannot silently disable the backstop; once the row has shown the job running, the coordinator latches that and never re-dispatches it.

• Worker spawn-retry verdict. The coordinator sends a spawn-retry budget on job.reroute (spawnRetry: { maxAttempts, backoffMs }). A worker retries a failed agent spawn within that budget, and relays each scaler.failed for the job back to the owning coordinator over the scaler-event channel with a final verdict:

  • final: true — the worker spent its last attempt and gave the job back. The coordinator runs the same cancel-then-re-dispatch immediately, without waiting the window out. The two layers are idempotent: whichever fires first removes the tracking entry, so the other is a no-op.
  • final: false — the worker is retrying. The coordinator restarts the spawn window for one more attempt plus the backoff it told the worker to wait. It treats the maxAttempts-th such relay as final, so a worker that never says final cannot hold the job forever.
  • no verdict — a repeated report of one failed spawn. The coordinator leaves the window it armed at the reroute running unchanged.

  The re-dispatch appends the failed peer to triedConnections so routing never re-selects it.

The spawn window, the reroute ACK timeout, the max-hop limit, and the worker’s spawn-retry budget are per-org tunables (reroute_spawn_window_ms, reroute_ack_timeout_ms, reroute_max_hops, reroute_spawn_max_attempts, reroute_spawn_retry_backoff_ms) with cluster-wide defaults; operators change them with kici-admin org-settings reroute.

Raft consensus

Purpose

KiCI uses a minimal Raft implementation for leader election only (no log replication). The Raft leader handles cluster-wide operations that must run on exactly one node:

• Orphan recovery: Detecting and finalizing runs from crashed coordinators
• Cron scheduling: Evaluating cron-triggered workflows to prevent duplicate firings across cluster nodes
• Event retry scanning: Re-dispatching events whose earlier delivery failed, under a lease so only one node retries a given event
• Recovery and ACK-deadline sweeps: Every 10s, failing any run stuck past its recovery deadline and any dispatch past its ACK deadline — this closes the window where a crashed coordinator’s in-memory timers were lost
• Pending-scale sweeping: Reconciling scale requests left pending by a crashed node
• Attestation retry: Re-attempting build attestations that failed to sign or publish
• Signing-key generation: Only the leader mints a fresh provenance signing key when none exists yet, so concurrent nodes cannot race two keys into existence

State machine

follower -> candidate -> leader
    ^           |            |
    |           v            |
    +--- (higher term) <----+

• Follower: Default state. Listens for leader heartbeats and vote requests. Starts election if no heartbeat received within election timeout.
• Candidate: Increments term, votes for self, requests votes from peers. Becomes leader with majority.
• Leader: Sends periodic heartbeats (every 2s), runs leader-only services.

Election parameters

Parameter	Default	Description
Election timeout	5-10s	Randomized jitter prevents simultaneous elections. WAN-appropriate.
Leader heartbeat	2s	Raft-specific heartbeat for leader liveness detection (separate from 30s inventory heartbeat).

Dormant mode

When the peer registry has 0 connected peers, the Raft node self-elects immediately. This makes single-orchestrator deployments trivially self-elected leaders without any special-casing. If peers connect later, normal election resumes.

Leaving the cluster

A coordinator that shuts down stops its Raft node before it disconnects from its peers. The stop is final. The node gives up leadership, so its leader-only services stop with it. It ignores every election message after the stop and grants no vote. So a peer.leaving from a sibling that stops at the same time cannot elect it again while it shuts down, and neither can its own peers dropping away. The remaining coordinators elect a new leader among themselves. The node’s last save to the shared raft_state row does not overwrite a newer term that they have already written.

Forgetting a departed peer

Each coordinator keeps its own live peer registry, and nothing removes a disconnected peer from it. A coordinator that left the cluster for good therefore stays in every registry as a disconnected coordinator peer. kici-admin peer forget <instance-id> drops it (packages/orchestrator/src/cluster/peer-forget.ts).

The coordinator that receives the request checks the peer on its own registry first (forgetDepartedPeer()) and keeps it, with the reason, when:

• the peer is connected (connected), because its next heartbeat registers it again.
• the peer was heard from inside its liveness window (recent), because it may be partitioned rather than gone. The window is the longer of the peer stale window and the reroute flap grace. A peer that adopted event-scaler provisions gets the backstop’s own flap grace instead.
• forgetting the last known coordinator peer would switch the event-provision backstop back on, and the request did not acknowledge that (acknowledgement-required).

A peer kept for one of these reasons is not fanned out. Otherwise the coordinator sends peer.forget.request to every connected sibling coordinator (forgetAcrossCoordinators()). Each sibling runs the same checks on its own registry and answers with peer.forget.response. A sibling that does not answer within the timeout is reported as error. Forgetting never touches the peer’s credential (that is peer revoke), so a forgotten peer that connects again is registered again as usual.

State persistence

Raft state (currentTerm, votedFor, leaderId) is persisted to the raft_state PostgreSQL table via upsert. Persistence uses fire-and-forget (.catch()) to avoid blocking election transitions on DB writes.

Webhook secret management

Architecture

Webhook secrets live exclusively on the orchestrator side, stored in the orchestrator’s shared PostgreSQL database (scoped_secrets table, read by PgSecretStore). The Platform tier never holds or caches webhook signing material — it asks the matching orchestrator to verify each inbound webhook over the chunked relay protocol.

Flow

1. The provider POSTs a webhook to the Platform relay with the signature header (e.g. X-Hub-Signature-256 for GitHub).
2. Platform routes the delivery to the orchestrators registered for the routing key and forwards the raw body + headers over the chunked relay protocol.
3. The orchestrator’s verifyInboundWebhook dispatcher loads the active secret(s) for the routing key from PgSecretStore on demand and checks the signature itself.
4. The orchestrator returns a verify-result over the same relay channel. Platform accepts or rejects the delivery based on the orchestrator’s answer.

Because verification runs on the orchestrator, secrets never leave customer infrastructure and Platform stays stateless with respect to webhook signing material.

Dual-secret rotation

The orchestrator supports multiple active secrets per routing key. During rotation:

1. Add a new secret row (old secret remains active).
2. Update the provider to use the new secret.
3. Wait for old-secret deliveries to drain.
4. Delete or expire the old secret row.

The verifier checks every active secret for the routing key, so old and new signatures both validate during the overlap window.

source.register decoupling

The source.register message the orchestrator sends to Platform on (re)connect carries only routing keys — no webhook secrets. Platform uses the routing-key list to map incoming deliveries to the right orchestrator and then asks that orchestrator to perform verification. This keeps connection-establishment messages free of secret material and lets secrets update independently of registration. Platform’s UnknownSourceCache short-circuits deliveries whose routing key matches no registered orchestrator, with negative caching (5-minute TTL) to avoid hammering the registry on bogus traffic.

Orphan recovery

When a coordinator crashes mid-run, its jobs may be stuck. The Raft leader runs periodic orphan recovery:

1. Scan: Find execution_runs in “running” state with stale timestamps (> 5 minutes)
2. Check coordinator: Look up the run’s routing_key in the peer registry. If a connected peer shares the routing key, the coordinator may still be alive — skip.
3. Check progress: Look at the run’s jobs for any heartbeat, start, or completion inside the stale window. A run that is still moving is not orphaned, whatever the peer registry says — skip. This is what keeps a healthy long run alive on a single-node orchestrator, where step 2 can never vouch for the coordinator because there are no peers. A genuinely crashed coordinator stops producing all three signals, so a real orphan becomes eligible within one window.
4. Check jobs: For each non-terminal job, check the heartbeat timestamp. Mark jobs with stale heartbeats (> 3 minutes) as failed.
5. Finalize: When all jobs are terminal, compute overall run status (failed if any failed, cancelled if any cancelled, success otherwise).

The Raft leader guard (if (!raft.isLeader()) return) prevents multiple orchestrators from racing to recover the same run.

Failure modes

Peer crash mid-job

The peer’s agent continues executing until it detects the connection loss. The coordinator marks the peer as disconnected in the peer registry. Jobs dispatched to that peer are considered failed when the peer disconnects.

Coordinator crash mid-run

Peers that have accepted rerouted jobs complete their jobs independently. The Raft leader detects the orphaned run during periodic recovery and finalizes it based on job terminal states.

Platform relay down

Direct peer connections continue functioning. Orchestrators that were discovered via the Platform matchmaker maintain their existing peer connections. New peer discovery is paused until the Platform connection recovers.

Network partition

If an orchestrator loses connectivity to peers, it continues processing webhooks and dispatching to local agents. Jobs that require peer rerouting fail with “No orchestrator in cluster has matching agents.” When connectivity recovers, peer connections are re-established via auto-reconnect.

Log routing

Pool validation

Multi-orchestrator pools recommend shared S3 log storage for consistent log access across all pool members. When a second orchestrator joins a pool (same orgId:routingKey), the Platform checks whether all pool members declare s3LogAccess: true in their source.register message. If there is a mismatch (some with S3, some without), the Platform logs a warning but allows the connection — this supports the coordinator/worker topology where the coordinator has S3 access while workers may not.

If all sources in a source.register message are rejected for other reasons and the rejections mention S3 log storage, the Platform closes the connection with WS close code 4010 as a safety fallback. In practice, this path is not triggered because pool validation always allows mixed S3 configurations.

Single-orchestrator pools can still use filesystem log storage (no S3 required).

The s3LogAccess field is sent in the source.register message (not just peer heartbeats), allowing the Platform to check pool membership at connection time.

Capability-based routing

Log routing is capability-based:

• Orchestrators that have S3 access (indicated via s3LogAccess in source.register and capabilities.s3LogAccess in peer heartbeats) write logs directly to S3
• Peers without S3 access route logs through the coordinator

Operators can override log routing by configuring S3 access on all orchestrators in the cluster.

Cluster identity

cluster_id (split-brain prevention)

Each cluster has a unique cluster_id (UUID) stored in the cluster_meta database table, auto-generated during the first migration. This identity prevents split-brain scenarios in geo-distributed deployments where different orchestrators might share the same Platform pool but point to different databases or S3 buckets.

On startup, each orchestrator:

1. Reads cluster_id from the cluster_meta table
2. If S3 storage is configured, checks for a .kici-cluster-id sentinel file in the S3 bucket (under the configured KICI_STORAGE_PREFIX if set, otherwise at the bucket root)
3. If the sentinel exists, validates that the S3 cluster_id matches the DB cluster_id
4. If they differ, the orchestrator refuses to start (different logical cluster despite same Platform pool)
5. If the sentinel is missing (first orchestrator to use this bucket+prefix), writes the sentinel

This allows different connection strings for the same logical resource (e.g., VPC endpoint vs public endpoint vs WireGuard IP) while preventing accidental bucket/DB sharing across distinct clusters.

Sharing one bucket across multiple clusters: Each cluster can use the same physical S3 bucket as long as it sets a distinct KICI_STORAGE_PREFIX. The sentinel lives at <prefix>/.kici-cluster-id, so two clusters with prefixes cluster-a/ and cluster-b/ get independent sentinels (cluster-a/.kici-cluster-id and cluster-b/.kici-cluster-id) and never collide. With no prefix, the sentinel is at the bucket root and the bucket is reserved for a single cluster.

Operational note: If you regenerate the DB (e.g. kici-admin db fresh) without also clearing or rewriting the sentinel, the orchestrator will refuse to start with Cluster identity mismatch: .... The error message includes the full s3://<bucket>/<prefix>/.kici-cluster-id path and suggests setting KICI_STORAGE_PREFIX to isolate the new cluster’s sentinel. The supported recovery is to either (a) restore the matching DB, (b) delete the sentinel object so the orch rewrites it from the new DB on next boot, or (c) point the orch at a fresh bucket+prefix combination.

Multiple clusters in one org sharing a source (unsupported)

KiCI’s design assumes one cluster per org for a given routing key. The Platform-side routing pool is keyed by (orgId, routingKey) only — there is no notion of cluster_id on the Platform tier. If two distinct clusters in the same org register the same source (e.g., the same GitHub App installation), every orchestrator from both clusters lands in the same Platform pool. The resulting behavior is incoherent and is not a supported deployment topology:

• Webhook routing splits unpredictably. Least-loaded routing picks one orchestrator from the merged pool per delivery. A given delivery may land on cluster A or cluster B, depending on momentary job counts and round-robin tie-breaking. The receiving orchestrator becomes the run coordinator and creates the run row in its own orchestrator DB; the other cluster never sees that delivery.
• Peer discovery cross-fires. Peer discovery matches on routing-key prefix only, so when a cluster B orchestrator registers, cluster A orchestrators are told about it and attempt P2P connections. The ECDH handshake completes, but mutual authentication then fails because cluster B’s peer-credential table has no entry for cluster A’s instance ID (and vice versa). The result is peer-discovery noise that never produces healthy peer links. A coordinator keeps its credential file through such a rejection, because its own database still holds the credential as valid.
• Webhook secrets must be duplicated. Each orchestrator verifies signatures locally against its own scoped-secret store. Whichever cluster gets routed must already hold the source’s signing material; otherwise signature verification fails inside the chosen coordinator and the delivery is rejected.
• Dashboard becomes split-view. The dashboard run list mirrors lifecycle events from every connected orchestrator, so runs from both clusters appear mixed together. Run-detail and log proxy queries route via least-loaded selection, so a request for a run that lives in cluster A’s DB can land on a cluster B orchestrator and return “not found”.
• Cron / orphan recovery double-fires. Each cluster’s Raft leader independently evaluates schedules and orphan recovery on its own DB, so any cron registration loaded into both clusters fires twice per tick, and orphan-run finalization decisions diverge between the two clusters.

There is no rejection or warning today — a second cluster’s source.register is accepted on the same path as a new orchestrator joining the first cluster (the only deduplication is the composite key (routing_key, orchestrator_connection_id)). The S3 sentinel and DB cluster_id checks above prevent two clusters from accidentally sharing storage, but they cannot detect the orthogonal mistake of two clusters sharing the same source.

If you genuinely need two independent orchestrator deployments listening to the same source — for example, a blue/green migration — the supported model is: keep one cluster on the source at a time, drain it, then switch. Running both simultaneously will exhibit the symptoms above.

Join token bootstrap

Join flow

New orchestrators can join an existing cluster with a join token, without manual config copying. kici-admin join uses join protocol v2: the join secret stays on the joining host, and the existing orchestrator seals the configuration bundle to a one-time key of that host.

Token format: kici_join_v1.<base64url_routing>.<hex_secret>

• Routing part (cleartext): { orgId, routingKey, expiry, role }. The Platform reads it to check the org and choose the target pool. The existing orchestrator uses it to find candidate token rows. It is a lookup key only: the role and routing key a join gets come from the stored row.
• Secret part: never sent. Its SHA-256 is the value the join_tokens table stores, and it is the root of every join key.

Key schedule. Both sides derive three keys from the SHA-256 of the secret with HKDF-SHA256: a joiner proof key, a server proof key, and a binding key. The joiner sends the routing part, a one-time X25519 public key, a random nonce, and an HMAC over a transcript of those three values. The existing orchestrator checks that proof against each candidate row and claims the matching row. It answers with its own one-time X25519 public key, its own nonce, an HMAC over the extended transcript, and the sealed bundle. The bundle key is HKDF over the X25519 agreement, salted with the binding key. The bundle is AES-256-GCM with the transcript as associated data. The joiner checks the server proof before it opens the bundle, and writes nothing when the proof fails.

A relay sees the routing part, both public keys, both nonces, both proofs and the ciphertext. None of these lets it read the bundle, forge one the joiner accepts, or join with the token.

The bundle carries the cluster database URL, the object storage configuration, the cluster’s secrets key (secrets.key from the shared configuration, else the answering orchestrator’s own KICI_SECRET_KEY), and the cluster ID.

Transport modes

Platform relay mode:

New orch --[join.request]--> Platform --[relay]--> Existing orch
New orch <--[join.response]-- Platform <--[relay]-- Existing orch

The joining host authenticates to the Platform with an API key of the org, as an orchestrator connection does. The Platform checks that the routing part names that org, then relays the join.request to an orchestrator in the target pool. When the pool is empty, the Platform answers no_target. The existing orchestrator answers through the relay. The Platform uses messageId correlation to route the response to the correct joiner when multiple join requests are in flight. The Platform refuses a join.request that carries the join token itself (join_protocol_v1_removed) and never forwards it.

Direct peer mode:

New orch --[POST /api/v1/cluster/join]--> Existing orch
New orch <--[sealed config bundle]------- Existing orch

Same protocol, but the new orchestrator posts the join.request to an existing orchestrator’s REST API instead of going through the Platform relay. Useful for independent setups without Platform connectivity. The endpoint answers a request body that carries the join token with 426 and Upgrade: kici-join-v2.

Token lifecycle

1. Operator creates a join token via POST /api/v1/admin/join-tokens or kici-admin peer create-token
2. The SHA-256 of the token secret is stored in the join_tokens DB table with expiry
3. New orchestrator runs kici-admin join --token TOKEN --platform URL or --peer URL
4. Existing orchestrator finds the row by the routing part, checks the proof, and claims the row
5. The claim marks the token consumed. The bootstrap join can repeat with the same token until it expires; a token consumed by a peer in token mode is refused to every other instance
6. New orchestrator checks the server proof, opens the bundle, and writes ./kici-orchestrator.env, the env file kici-admin orchestrator install --env-file consumes

Peer credential issuance

After successful join token validation, the coordinator issues a persistent session credential to the new peer. The credential is stored in the peer_credentials database table and saved to the peer’s local credential file (~/.kici/peer-credential). Subsequent connections prove the credential (a mutual-v2 proof over the handshake transcript) instead of the join token. Credentials can be revoked via kici-admin peer revoke.

A coordinator without a join token issues its own credential instead, the first time it connects to a peer. It writes the credential hash to peer_credentials (marked as self-issued) and the credential to its local file. Then it proves possession like any joined peer. It retires the self-issued credential its previous run left in the same file, unless that instance still has a live heartbeat. It does not re-issue a credential that an operator revoked for its instance ID; a coordinator that restarts under a new instance ID issues a credential for the new ID. Every coordinator shares the cluster database, so issuing its own credential needs no access beyond what kici-admin peer create-token needs.

Data flow: webhook to check run update

See also

• Multi-orchestrator clustering (operator guide) — deployment recipes, configuration reference, join tokens, and troubleshooting for running a cluster.
• Scaling ceilings — the peer-heartbeat and Platform-routing tripwires, and the growth conditions that should trigger reworking them.