Multi-orchestrator clustering

KiCI supports running multiple orchestrators in a cluster. Clustering enables high availability (HA) through redundancy, cross-architecture job routing (e.g., x64 + ARM64 agents), and dedicated coordinator topologies for large deployments.

Cluster components (Raft consensus, peer registry, health endpoints) are always initialized. A single-orchestrator deployment works unchanged — Raft dormant mode self-elects as leader immediately with zero overhead. No extra configuration is needed for single-orch deployments.

Overview

In a cluster, each orchestrator is a full-featured instance that can both receive webhooks and execute jobs. When a webhook arrives, the receiving orchestrator becomes the run coordinator for that webhook event. The coordinator matches triggers against the lock file, claims jobs it can dispatch to its own local agents, and reroutes remaining jobs to peers with matching capacity.

Key capabilities:

• HA pair — two identical orchestrators behind the Platform round-robin for redundancy
• Cross-architecture pools — x64 and ARM64 orchestrators with different scalers, same routing key
• Dedicated coordinator — one orchestrator with no scalers (coordinator-only) handles webhook processing while others handle execution
• Automatic peer discovery — in Platform/hybrid modes, the Platform matchmaker tells orchestrators about peers
• Raft leader election — one orchestrator is elected leader for cluster-wide operations like orphan run recovery

How many clusters?

Most orgs need one cluster. A single cluster spans architectures and hardware shapes through multiple scalers — x64 containers, ARM64 containers, bare-metal, Firecracker, GPU pools — and routes each job to a matching scaler by its runs-on labels. To add capacity or a new agent type (an ARM64 pool, a GPU pool), add a scaler, not a cluster: the x64 + ARM64 pool recipe below runs one cluster across two architectures, and the auto-scaler overview covers the available backends.

High availability also lives inside one cluster: redundancy comes from running multiple coordinators that share one PostgreSQL (the HA pair recipe below), not from standing up more clusters.

Reserve a separate cluster for a genuine isolation boundary — a distinct team that needs its own dashboard write policy, environments, secrets, and sources. A separate cluster is not the default for every environment, region, or architecture.

Every additional cluster carries real setup cost: its own sources and webhook registration, its own join-token bootstrap, its own (possibly divergent) configuration, and a separate dashboard surface to operate. Prefer fewer clusters with more scalers until an isolation boundary forces a split.

Prerequisites

Shared API key (coordinators only)

All coordinator orchestrators in a cluster authenticate to the Platform using the same API key (KICI_PLATFORM_TOKEN). This is by design — a cluster is a single operator’s deployment, and the API key is scoped to that operator’s organization. Each coordinator connects independently to the Platform, but they all belong to the same org and share webhook sources, runs, and dashboard data.

Workers do not need a Platform token. Workers connect only to their coordinator via P2P WebSocket (KICI_CLUSTER_COORDINATOR_URL) and never talk to the Platform relay. They have no database, no S3 credentials, and no API key — see coordinator/worker deployment for details.

Different operators (customers) must use separate API keys and organizations. Each API key is bound to a single organization on the Platform, so cross-tenant traffic is rejected at the relay before it reaches any orchestrator.

Shared PostgreSQL (mandatory)

All orchestrators in a cluster must share the same PostgreSQL database. This is a hard requirement — join tokens, peer credentials, Raft state, execution tracking, webhook secrets, cluster metadata, and the dispatch queue all live in this shared database.

The shared database ensures:

• Join token consumption is atomic across coordinators (DB transactions prevent races)
• Peer credentials are globally visible, so any coordinator can validate a reconnecting peer
• Cluster identity (cluster_id in cluster_meta) is consistent — at startup, each coordinator validates it reads the same cluster_id, preventing accidental misconfiguration where separate clusters point at the same database. When S3 storage is configured, the validation also checks a <prefix>/.kici-cluster-id sentinel object so two clusters can safely share a physical bucket if they use distinct KICI_STORAGE_PREFIX values. See cluster identity in multi-orchestrator design for the full sentinel contract
• Credential tracking records which coordinator last validated each peer (last_validated_by), providing operational visibility into connection routing

Do not run separate PostgreSQL instances per orchestrator in a cluster. This will cause split-brain: each orchestrator would generate a different cluster_id, tokens consumed on one would not be visible to others, and peer credentials would be siloed.

Peer authentication (join tokens)

Peer-to-peer WebSocket connections use an ECDH key exchange followed by join token or credential authentication. The first orchestrator (coordinator) starts normally. Additional peers join using a one-time join token that authenticates them and issues a persistent credential for subsequent connections.

Network connectivity

Orchestrators need to reach each other via WebSocket for direct peer connections. Set KICI_CLUSTER_ADDRESS to a reachable host:port for each orchestrator. Without a reachable address, peers cannot establish direct connections.

In Platform/hybrid modes, orchestrators that cannot reach each other directly fall back to relay through the Platform tier.

For the full outbound allowlist and inbound surface across all deployment modes, see Network requirements.

Configuration reference

Cluster configuration uses the KICI_CLUSTER_* environment variable prefix.

Environment Variable	Default	Required	Description
KICI_CLUSTER_JOIN_TOKEN	—	First join only	One-time join token for authenticating with the cluster. Used only on the first connection; a persistent credential is issued after successful authentication.
KICI_CLUSTER_CREDENTIAL_FILE	~/.kici/peer-credential	—	Path to store/load the persistent peer credential. After first join, the orchestrator uses this credential for all subsequent connections.
KICI_CLUSTER_INSTANCE_ID	random UUID	Recommended	Unique identifier for this orchestrator instance. Auto-generated if not set, which gives a restarted orchestrator a new id and orphans the rows its previous boot wrote — set a stable value on every orchestrator, see recovery is scoped to the owning instance.
KICI_CLUSTER_ADDRESS	—	When peers set	This orchestrator’s reachable address (e.g., ws://10.0.0.1:4000). Required when KICI_CLUSTER_PEERS is set.
KICI_CLUSTER_PEERS	—	Multi-orch independent	Comma-separated list of peer addresses (e.g., ws://10.0.0.2:4000,ws://10.0.0.3:4000). Only needed for multi-orchestrator independent mode (Platform/hybrid uses automatic peer discovery).
KICI_CLUSTER_RAFT_ELECTION_TIMEOUT_MIN_MS	5000	—	Minimum Raft election timeout (ms).
KICI_CLUSTER_RAFT_ELECTION_TIMEOUT_MAX_MS	10000	—	Maximum Raft election timeout (ms).
KICI_CLUSTER_RAFT_HEARTBEAT_MS	2000	—	Raft leader heartbeat interval (ms).
KICI_CLUSTER_PEER_HEARTBEAT_INTERVAL_MS	30000	—	Peer inventory heartbeat interval (ms).
KICI_CLUSTER_INSTANCE_HEARTBEAT_MS	10000	—	How often this orchestrator records its own liveness in the shared cluster_instances table (ms). Every recovery decision reads it, so raising it widens the window in which a crashed coordinator’s jobs stay untouched. See recovery is scoped to the owning instance.
KICI_CLUSTER_PEER_MAX_RECONNECT_DELAY_MS	60000	—	Maximum delay between peer reconnect attempts (ms).
KICI_CLUSTER_ROLE	coordinator	For workers	Cluster role: coordinator (default, full orchestrator) or worker (delegated execution only). See Coordinator/worker deployment.
KICI_CLUSTER_COORDINATOR_URL	—	For workers	WebSocket URL of the coordinator’s peer endpoint (e.g., ws://coordinator:4000/ws/peer). Required when KICI_CLUSTER_ROLE=worker.
KICI_CLUSTER_PEER_STALE_TIMEOUT_MS	60000	—	Timeout in ms after which a peer with no heartbeat is considered stale.
KICI_CLUSTER_TRUSTED_PROXIES	—	Behind reverse proxy	Comma-separated list of trusted proxy IPs or CIDR ranges (e.g., 10.0.0.0/8,172.16.0.0/12). When set, the peer handler extracts the real client IP from X-Forwarded-For instead of using the socket IP. Required for correct rate limiting when orchestrators are behind a load balancer or reverse proxy.

Mode-specific requirements

• Single-orchestrator (any mode): No cluster env vars needed. Cluster components initialize in dormant mode automatically.
• Multi-orchestrator Platform/hybrid mode: The first orchestrator starts normally. Additional peers need a join token created by kici-admin peer create-token. The Platform matchmaker handles peer discovery automatically.
• Multi-orchestrator independent mode: KICI_CLUSTER_JOIN_TOKEN (for peers), KICI_CLUSTER_ADDRESS, and KICI_CLUSTER_PEERS are required because there is no Platform matchmaker for peer discovery.

Peer authentication flow

Peer authentication uses ECDH (X25519) key exchange to establish an encrypted channel before any credentials are transmitted.

First join (with token)

1. Start the coordinator — the first orchestrator in the cluster starts normally with KICI_SECRET_KEY set
2. Create a join token on the coordinator:

   kici-admin peer create-token --role coordinator
   # or for a worker peer:
   kici-admin peer create-token --role worker

3. Start the peer with the join token:

   KICI_CLUSTER_JOIN_TOKEN=kici_join_v1.xxx.yyy

4. ECDH handshake — the peer and coordinator exchange ephemeral X25519 public keys via peer.hello / peer.hello.response messages
5. Encrypted auth — the peer sends the join token in an encrypted peer.auth.request. The coordinator validates the token and responds with an encrypted peer.auth.response containing a session credential
6. Credential persistence — the peer saves the issued credential to KICI_CLUSTER_CREDENTIAL_FILE (default: ~/.kici/peer-credential)
7. Token bound to the peer — the join token is marked consumed and recorded against the joining peer’s instance ID. Until the token expires, the same peer instance may present it again to re-acquire a credential (self-healing rejoin after a transient outage); a different instance cannot reuse it

Subsequent connections (with credential)

After the first join, the orchestrator uses its persisted credential file for all subsequent connections. No join token is needed:

1. The orchestrator loads the credential from KICI_CLUSTER_CREDENTIAL_FILE
2. ECDH handshake establishes an encrypted channel
3. The orchestrator sends an HMAC proof of the credential (never the credential itself)
4. The coordinator verifies the HMAC proof using timingSafeEqual
5. Connection authenticated — the peer resumes normal operation

Security properties

• No cleartext auth material — all authentication happens over the ECDH-encrypted channel
• Instance-bound tokens — a join token is consumed on first use and bound to the joining peer’s instance ID; only that same instance may re-present it (until expiry) to self-heal, so a returning peer recovers without a full cluster redeploy while a leaked token cannot be replayed by a different instance
• HMAC credential proof — credentials are never sent over the wire; only an HMAC proof is transmitted
• Rate limiting — failed authentication attempts are rate-limited (5 attempts per IP within 60 seconds)
• Post-auth encryption — all subsequent messages (heartbeats, reroutes, Raft) use the ECDH-derived session key

Deployment recipes

HA pair

Two identical orchestrators sharing the same PostgreSQL, scalers, and routing key. Provides redundancy: if one goes down, the other continues processing webhooks.

                 +---> Orchestrator A (coordinator)
  Platform ----+     |
  (round   |     |
   robin)  +-----+
                 |
                 +---> Orchestrator B (peer)
                       joins via token

Step 1: Start Orchestrator A (coordinator):

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_ADDRESS=ws://orchestrator-a:4000
KICI_DATABASE_URL=postgres://user:pass@shared-db:5432/kici
KICI_SCALER_CONFIG_PATH=/etc/kici/scalers.yaml

Step 2: Create a join token on Orchestrator A:

kici-admin peer create-token --role coordinator

Step 3: Start Orchestrator B with the token:

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_JOIN_TOKEN=kici_join_v1.xxx.yyy
KICI_CLUSTER_ADDRESS=ws://orchestrator-b:4000
KICI_DATABASE_URL=postgres://user:pass@shared-db:5432/kici
KICI_SCALER_CONFIG_PATH=/etc/kici/scalers.yaml

After the first successful connection, Orchestrator B saves its credential and no longer needs the join token. Subsequent restarts use the persisted credential automatically.

x64 + ARM64 pool

Two orchestrators with different scalers targeting different architectures. Same routing key so both receive webhooks. Jobs are automatically routed to the orchestrator with agents matching the job’s runs-on labels.

                 +---> Orchestrator A
  Platform ----+     |     scalers: x64 containers
  (round   |     |     agents: [self-hosted, linux, x64]
   robin)  +-----+
                 |
                 +---> Orchestrator B
                       scalers: ARM64 containers
                       agents: [self-hosted, linux, arm64]

Orchestrator A (x64, coordinator):

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_ADDRESS=ws://orch-x64:4000
KICI_SCALER_CONFIG_PATH=/etc/kici/scalers-x64.yaml

Orchestrator B (ARM64, peer — first start):

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_JOIN_TOKEN=<token-from-orch-a>
KICI_CLUSTER_ADDRESS=ws://orch-arm64:4000
KICI_SCALER_CONFIG_PATH=/etc/kici/scalers-arm64.yaml

When a webhook arrives at Orchestrator A and the job requires arm64 labels, the coordinator reroutes it to Orchestrator B (which has ARM64 agents). The coordinator still handles all check run reporting.

Dedicated coordinator + workers

One orchestrator with no scalers acts as the coordinator: it processes webhooks, matches triggers, creates check runs, and reroutes all jobs to worker orchestrators. Workers have scalers and agents but delegate check run reporting to the coordinator.

                 +---> Coordinator (no scalers)
  Platform ----+     |     handles: webhook processing, check runs
  (round   |     |
   robin)  +-----+---> Worker A (x64 scalers)
                 |     handles: job execution only
                 |
                 +---> Worker B (ARM64 scalers)
                       handles: job execution only

Coordinator:

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_ADDRESS=ws://coordinator:4000
# No KICI_SCALER_CONFIG_PATH -- this orchestrator has no agents

Workers (first start):

KICI_MODE=platform
KICI_SECRET_KEY=<64-char-hex-key>
KICI_CLUSTER_JOIN_TOKEN=<token-from-coordinator>
KICI_CLUSTER_ADDRESS=ws://worker-a:4000  # or worker-b
KICI_SCALER_CONFIG_PATH=/etc/kici/scalers.yaml

Create separate join tokens for each worker using kici-admin peer create-token --role worker.

In this topology, the coordinator always reroutes jobs since it has no local agents. Workers accept rerouted jobs and dispatch to their agents. The coordinator tracks job progress from workers and updates GitHub check runs.

Autoscaling in a cluster

Every orchestrator loads its own scaler config file and builds its own backends from it. Nothing distributes that file, so each one decides for itself what it can provision. The state behind those decisions is split: some of it is shared through the cluster database, and the rest stays in the process that holds it.

What is shared

These live in the shared orchestrator database, so any coordinator can read them:

State	Why it is shared
Pending provisioning claims	A claim code minted by one coordinator is redeemed by whichever coordinator the new agent reaches.
Spawn records	They carry the scaler name, the labels, the taint labels, and the provisioning targets, so a coordinator with no config for that scaler can still adopt the agent and tear it down.
Ephemeral agent tokens	The agent registers on whichever coordinator it reached.
Agent-to-job correlation	Lifecycle events stay attached to the run when another coordinator takes the agent.
Resource reservations	Each row is stamped with the coordinator holding it.
Reaper windows	They are cluster settings, read fresh on each sweep.
Precursor job results	A build, init, or dynamic-eval result is written on the shared job row, so the coordinator that dispatched the job reads it back when another coordinator’s agent ran it.
Registration windows	The run row names the coordinator that still has jobs to register for it, so no sibling finalizes the run early. See runs that span coordinators.

What stays local

• The scaler config file and its backends. A coordinator with no KICI_SCALER_CONFIG_PATH builds no scaler at all, which is exactly what the dedicated-coordinator recipe above wants. One exception: if the cluster runs an event scaler, give every coordinator behind the shared endpoint a config file, even an empty one — see event scaler high availability for why a coordinator without one cannot serve a provisioned agent.
• The caps of a local backend. maxAgents on a container, bare-metal, or firecracker scaler bounds that host, and so do globalMaxAgents and the CPU and memory totals. An event scaler is the exception: its maxAgents is counted across the cluster. See caps in a cluster.
• The warm pool. Each orchestrator keeps its own.
• The absence clock behind the stranded-provision sweep. It lives on the leader and restarts when leadership moves.

Recovery is scoped to the owning instance

On boot, an orchestrator recovers only the rows it owns, plus the rows whose owning coordinator is no longer alive. It leaves every other row alone. This applies to two planes:

• The scaler plane — spawn records and resource reservations. Scoping stops an orchestrator counting a peer’s reservations against its own caps and refusing spawns it has capacity for.
• The dispatch plane — in-flight jobs. Scoping stops a booting orchestrator starting recovery timers for jobs the other coordinators’ agents are running. An unscoped boot marked those jobs recovering and, once the recovery window passed, failed them — while the agents ran them to completion.

Set KICI_CLUSTER_INSTANCE_ID to a stable value on every orchestrator in a cluster. An auto-generated id is new on every boot, so a restarted orchestrator does not recognise the rows its own previous boot wrote.

Coordinator liveness

Each orchestrator writes its own liveness into a shared cluster_instances table every KICI_CLUSTER_INSTANCE_HEARTBEAT_MS (10 seconds by default). A coordinator counts as alive while its heartbeat is inside a grace window of the larger of the agent recovery window and six heartbeat intervals — 120 seconds at the defaults. A currently-connected peer also counts as alive, as a corroborating signal.

The heartbeat is in the database rather than derived from peer connections because a booting orchestrator has not connected to any peer yet, and boot is exactly when recovery runs. It needs no peer connectivity, so it behaves the same in a Raft cluster, a plain multi-coordinator deployment, and a standalone one.

A clean shutdown removes the row, so a coordinator stopped on purpose is recognised as gone immediately rather than after the grace window.

Runs that span coordinators

The dispatch queue is cluster-wide. An agent connected to any coordinator can claim a queued job, so the jobs of one run can execute on agents of several coordinators. Each agent reports to the coordinator it is connected to, which writes the job’s status to the shared execution_jobs row. Between coordinators, the shared row is the report. No configuration is needed.

Three mechanisms keep such a run correct:

• Precursor results travel on the job row. A workflow’s __build__, __init__, or dynamic-eval job produces a result the dispatching coordinator waits for before it dispatches the real jobs. When another coordinator’s agent ran that job, the result is written to the shared row and the waiting coordinator reads it back within a few seconds. It does not wait out its build timeout.
• Completion is decided from the shared rows. A coordinator finalizes a run only when every job row of the run is terminal, not only the jobs its own agents reported. The run’s status covers every job.
• A registration window blocks early completion. While a coordinator’s dispatch pipeline still has jobs to register for a run (the post-build jobs, the jobs of a deferred init), the run row names that coordinator as the holder of a registration window. Every other coordinator defers finalizing the run while that holder is alive. A build job that finishes on a sibling is therefore not read as “the run is complete”.

A holder that dies mid-window reads as no window. The stale-job detector finishes any run whose every job is terminal but whose finalizer never ran.

Rows with no recorded owner

A dispatch row written before its orchestrator recorded owners — during a rolling upgrade, for instance — has no owner recorded. An unrecorded owner reads as “unknown”, never as “not mine”. Such a row is skipped by recovery rather than claimed, and the orchestrator logs how many rows it skipped and why.

Skipping is not the same as losing the job. A row whose dispatch genuinely went nowhere is still reaped by the stale-job detector, which measures how long a dispatch has gone unacknowledged from when it was handed to an agent. Once every orchestrator in the cluster has restarted onto a version that records owners, no new rows have an unrecorded owner.

Event spawn records survive a lost instance id: any coordinator can adopt one when the agent registers, and the leader’s sweep tears down whatever stays unclaimed. Both paths match on backend_type = 'event', so a container, bare-metal, or firecracker spawn record whose owning instance never comes back is adopted by nobody and swept by nobody. Resource reservations behave the same way: a reservation row whose owning instance never comes back is read by nobody, and nothing deletes it. Both kinds of orphan cost table space only. Neither consumes capacity, because each orchestrator counts only the rows it owns, and the cluster-wide event-scaler cap counts event spawn records instead. Setting a stable KICI_CLUSTER_INSTANCE_ID is what stops either from accumulating across orchestrator churn.

Storage configuration (SharedConfig)

S3 recommendation for multi-orchestrator pools

Multi-orchestrator pools strongly recommend shared S3 log storage so any pool member can serve historical logs (the Platform’s leastLoaded() router can pick any orch in the pool to handle a dashboard request). When a second orchestrator joins a pool, the Platform compares its s3LogAccess flag against the existing members and logs a warning if they disagree, but does not reject the connection — this supports the coordinator/worker topology where the coordinator has S3 access and the workers do not (workers reroute jobs but do not serve logs directly).

The s3LogAccess flag is true whenever storage.type=s3 is configured (the same S3 backend underpins both the cache storage and the log storage). When it is false, the orchestrator falls back to a filesystem LogStorage rooted at ${webhookPayloadDir ?? '/var/lib/kici/cache'}/logs — logs and webhook payloads are still stored, just on the local disk of the orch that ingested the run. Source-tarball / dep-tarball caching is disabled in this mode (every run recompiles).

Single-orchestrator deployments can use filesystem storage with no penalty — the same orch ingests, executes, and serves logs.

The s3LogAccess field is sent in the source.register message (and in peer heartbeats), so the Platform records it per connection and surfaces it on the infrastructure page.

SharedConfig storage migration

Storage settings can be supplied either via env vars on each orchestrator or via the SharedConfig system in the database. Both paths land in the same storage.* config field.

Env vars and the SharedConfig fields they populate:

Env var	SharedConfig field
KICI_STORAGE_TYPE	storage.type
KICI_STORAGE_BUCKET	storage.bucket
KICI_STORAGE_PREFIX	storage.prefix
KICI_STORAGE_REGION	storage.region
KICI_STORAGE_ENDPOINT	storage.endpoint
KICI_STORAGE_EXTERNAL_ENDPOINT	storage.externalEndpoint
KICI_STORAGE_UPLOAD_ENDPOINT	storage.uploadEndpoint
KICI_STORAGE_FORCE_PATH_STYLE	storage.forcePathStyle
KICI_STORAGE_LOG_BUCKET	storage.logBucket

Migration path:

1. Move storage settings from env vars to the SharedConfig in the database
2. Remove the env vars from your deployment
3. Restart the orchestrators

The orchestrator bridges env vars into config.storage at startup. Once migrated to SharedConfig, all orchestrators in the cluster share the same storage configuration automatically.

Note: storage is a restart-required field. Changes to storage config detected during hot-reload emit a warning but are not applied until the orchestrator is restarted.

Cluster join tokens

Join tokens enable zero-knowledge cluster bootstrap. A new orchestrator can join an existing cluster with a single command — no manual config copying required.

Creating a join token

Via CLI (recommended):

kici-admin peer create-token --role coordinator
# or for a worker:
kici-admin peer create-token --role worker

Via API:

curl -X POST https://orchestrator:4000/api/v1/admin/join-tokens \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "orgId": "my-org",
    "routingKey": "github:12345",
    "role": "coordinator",
    "expiryMs": 3600000
  }'

Response:

{
  "token": "kici_join_v1.eyJvcmdJZCI6Im15LW9yZyIsInJvdXRpbmdLZXkiOiJnaXRodWI6MTIzNDUiLCJleHBpcnkiOjE3MTY5MjAwMDB9.a1b2c3d4...",
  "expiresAt": "2026-03-20T08:00:00.000Z"
}

Endpoint: POST /api/v1/admin/join-tokens

Field	Type	Required	Description
orgId	string	Yes	Organization ID for the target pool
routingKey	string	Yes	Routing key (e.g., github:12345)
role	string	No	Peer role: coordinator (default) or worker
expiryMs	number	No	Token validity duration in ms (default: 1h)

Requires token.manage RBAC permission.

Using a join token

Set the token as an environment variable before starting the orchestrator:

KICI_CLUSTER_JOIN_TOKEN=kici_join_v1.xxx.yyy

The orchestrator authenticates with the token on its first connection. After successful authentication, a persistent credential is issued and saved to the credential file. The token is consumed and bound to this peer’s instance ID; in normal operation the persisted credential is used from then on, so the token is no longer needed.

If a peer later loses its credential (a transient outage during a token rotation, or a deleted credential file), it falls back to the join token already in its environment. Because the token is reusable by the same instance until it expires, the coordinator accepts the reuse and issues a fresh credential — the peer self-heals and rejoins without operator action or a cluster redeploy. A token that has fully expired still requires a new one via kici-admin peer create-token.

Token security

• Tokens are instance-bound — consumed on first use and recorded against the joining peer’s instance ID. The same instance may re-present an unexpired token to self-heal; a different instance is rejected as already used
• Reuse is bounded by both the token’s expiry and the consuming instance ID, so a leaked consumed token cannot be replayed as a different peer
• Token hashes (not plaintext) are stored in the database
• Auth material is transmitted over an ECDH-encrypted channel (never in cleartext)
• Tokens carry a role (coordinator or worker) enforced at join time

Credential management

After the initial join, peers authenticate using persistent credentials stored in a local file.

Credential file

The credential file is stored at ~/.kici/peer-credential by default (configurable via KICI_CLUSTER_CREDENTIAL_FILE). It contains:

• Instance ID
• Credential hash
• Role (coordinator or worker)
• Coordinator URL
• Issued timestamp

The file is created with 0600 permissions (owner read/write only).

Managing peers

List active peers:

kici-admin peer list

Revoke a peer:

kici-admin peer revoke --instance-id <id>

The revoked peer’s credential is invalidated. The peer must re-join with a new token.

Revoke all peers:

kici-admin peer revoke-all --confirm

All peer credentials are invalidated. All peers must re-join with new tokens. Use this for emergency security responses.

Re-joining after revocation

If a peer’s credential is revoked:

1. Create a new join token: kici-admin peer create-token --role coordinator
2. Set the token on the revoked peer: KICI_CLUSTER_JOIN_TOKEN=<new-token>
3. Restart the peer
4. The peer authenticates with the new token and receives a new credential

Monitoring

Health endpoints

Three cluster HTTP endpoints are always mounted:

Endpoint	Description
GET /cluster/health	Overall cluster health: status (healthy/degraded/unhealthy), role, term, leader, peer count, agent count, active runs
GET /cluster/peers	Per-peer details: instance ID, connection state, agent count, draining status, capabilities
GET /cluster/runs	Active execution runs with job routing summary

Health status logic:

• healthy — Raft leader exists AND all peers connected (or single-node with no peers)
• degraded — Raft leader exists and majority of peers connected, but some peers disconnected
• unhealthy — no Raft leader OR fewer than majority of nodes connected

Example health check:

curl -s http://orchestrator:4000/cluster/health | jq .

{
  "status": "healthy",
  "instanceId": "orch-a-abc123",
  "role": "leader",
  "term": 3,
  "leaderId": "orch-a-abc123",
  "peerCount": 1,
  "connectedPeers": 1,
  "agentCount": 4,
  "activeRuns": 2
}

Prometheus metrics

Cluster operations emit standard Prometheus metrics alongside existing orchestrator metrics. Relevant metrics include peer connection counts, job reroute totals, Raft election events, and orphan recovery counts.

Webhook secret management

Webhook secrets live in the orchestrator’s shared PostgreSQL database under scoped_secrets (the encrypted PgSecretStore), keyed by source ID. The sources table holds the source’s metadata (routing key, provider config) and joins to the secret on __source__/<sourceId>. Use kici-admin source to manage them — never write to scoped_secrets directly.

How secrets work

1. Operators register sources via the CLI: kici-admin source add github --app-id <id> --webhook-secret <secret> --private-key @<path>. The CLI inserts a row into sources and writes the webhook secret into scoped_secrets via PgSecretStore.setSecret().
2. The orchestrator sends source.register to the Platform on connect, advertising the routing key (e.g., github:12345). The webhook secret is never sent over the wire — the message carries only the routing key and provider type.
3. When a webhook arrives at the Platform, the Platform sends a webhook.relay.start / webhook.relay.chunk sequence to the orchestrator that owns the routing key. The orchestrator reads the secret from PgSecretStore and verifies the HMAC signature locally, then ACKs back to the Platform with the verification outcome.
4. The Platform tier never holds webhook secret material — verification is delegated to the orchestrator on every inbound delivery.

Rotating a webhook secret

Each source carries exactly one active webhook secret. To rotate without dropping deliveries:

1. Update the provider first (e.g., GitHub App settings) to use the new secret. Provider-side rotation typically tolerates a brief window where in-flight deliveries still carry the old signature, but the new secret takes effect on the next delivery.
2. Update the orchestrator-side secret:

   kici-admin source update <routingKey> --webhook-secret <new-secret>

   This rewrites scoped_secrets in a single transaction. Inflight webhook.relay verifications may briefly fail during the swap; the provider will retry.
3. After the rotation settles, operate as normal. There is no second-row “two-secret” mode — the source carries one secret at a time.

For a coordinated zero-downtime rotation, drain webhook traffic at the load balancer for the few seconds between provider and orchestrator updates.

Rate limiting behind reverse proxies

Peer authentication rate limiting tracks failed attempts by IP address. When orchestrators sit behind a reverse proxy or load balancer, all connections appear to come from the proxy’s IP, which can cause legitimate peers to be rate-limited when a single bad actor triggers the limit.

Set KICI_CLUSTER_TRUSTED_PROXIES to the proxy’s IP or CIDR range so the peer handler extracts the real client IP from the X-Forwarded-For header:

KICI_CLUSTER_TRUSTED_PROXIES=10.0.0.0/8,172.16.0.0/12

Without this setting, rate limiting uses the socket IP (the proxy), which may incorrectly block all peers behind the same proxy.

Troubleshooting

Peer authentication failed

Symptom: Peer connections fail with “Invalid join token” or “Invalid credential” in logs.

Checks:

1. Join token expired — tokens expire after 1 hour by default. Create a new one with kici-admin peer create-token
2. Token already consumed — join tokens are one-time use. Create a new one for each peer
3. Credential revoked — if the credential was revoked via kici-admin peer revoke, the peer needs a new join token
4. Rate limited — after 5 failed auth attempts within 60 seconds, the IP is temporarily blocked. Wait and retry

Peers not connecting

Symptom: /cluster/peers shows 0 connected peers despite multiple orchestrators running.

Checks:

1. Verify KICI_CLUSTER_ADDRESS is set and reachable from peer orchestrators
2. In independent mode, verify KICI_CLUSTER_PEERS lists all peer addresses
3. Check firewall rules allow WebSocket connections on the orchestrator port
4. In Platform mode, verify both orchestrators connect to the same Platform relay and register the same routing key

Stale peers

Symptom: /cluster/peers shows peers as connected but their last heartbeat is old.

The peer heartbeat interval is 30 seconds by default. If heartbeats stop, the peer may have crashed or lost network connectivity. The peer will be marked as disconnected after the connection closes.

Orphan runs

Symptom: Execution runs stuck in “running” state after an orchestrator crash.

The Raft leader runs periodic orphan recovery (every 60 seconds). It detects runs whose coordinator orchestrator is no longer connected and finalizes them. If no leader is elected, orphan recovery cannot run — check /cluster/health to verify a leader exists.

Job routing limits

Rerouted jobs carry a hop counter to prevent infinite routing loops. If a job exceeds the maximum hop count, it fails instead of being rerouted again.

Limit	Default	kici-admin org-settings reroute flag	Description
Maximum hops	3	--max-hops	Jobs rerouted more than this many times are failed to prevent loops
ACK timeout	15s	--ack-timeout	Time for a peer to acknowledge receipt of a rerouted job
Spawn window	90s	--window	After a peer ACKs, how long to wait for progress before re-dispatching

Each default is cluster-wide; set a per-org override with kici-admin org-settings reroute set --org <id> --window <ms> --ack-timeout <ms> --max-hops <n>, and clear it with kici-admin org-settings reroute reset --org <id>.

Rerouted job stalls (spawn failure)

Symptom: a job rerouted to a peer never starts and the run sits pending.

A peer that accepts a reroute but then fails to spawn the agent (transient scaler error, image-pull failure, peer crash) does not strand the run. The coordinator arms a spawn window on every accepted reroute. When the window elapses with no progress, the coordinator first checks whether the job has actually started. A peer orchestrator shares this database and records the job’s status in it directly. The coordinator reads that status, leaves a running job alone, and disarms the window. Only a job that never started is cancelled on the original peer. The coordinator then re-dispatches it to another peer or a local backend, and fails it only if no backend can run it. A worker that detects the spawn failure directly reports it back, so recovery usually happens well before the window elapses. Raise --window for peers with legitimately slow agent startup; lower it to recover faster on flaky backends.

Jobs not rerouting

Symptom: Jobs fail with “No orchestrator in cluster has matching agents” even though a peer has agents.

Checks:

1. Verify the peer is connected (/cluster/peers)
2. Verify the peer’s agents have matching labels (labels are shared via heartbeat)
3. Verify the peer’s agents have available capacity (not at max concurrency)
4. Verify the peer is not draining

See also

• Multi-orchestrator architecture — coordinator/worker model, Raft consensus, peer communication, and the rerouting protocol behind this deployment guide.
• Coordinator/worker architecture — how the dedicated-coordinator topology splits webhook processing from job execution.