Dashboard, metrics & wire format

This page documents the WebSocket messages that flow on the agent↔orchestrator and orchestrator↔orchestrator (peer-to-peer) channels, plus the wire format invariants every tier obeys.

Concurrency protocol messages

These messages enable job-level concurrency control. When an agent discovers a job belongs to a concurrency group (from the workflow definition), it reports to the orchestrator, which decides the action.

Authoritative source: packages/engine/src/protocol/messages/orchestrator-agent.ts

job.concurrency.report

Direction: Agent -> Orchestrator

Agent reports that a job belongs to a concurrency group, requesting permission to proceed.

Field	Type	Required	Description
type	"job.concurrency.report"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
runId	string	Yes	Execution run ID
jobId	string	Yes	Job ID within the run
group	string	Yes	Concurrency group name

job.concurrency.ack

Direction: Orchestrator -> Agent

Orchestrator acknowledges concurrency report with action to take.

Field	Type	Required	Description
type	"job.concurrency.ack"	Yes	Message discriminator
requestId	string	Yes	Correlation ID from the report
action	enum	Yes	One of: proceed, wait, cancel
reason	string	No	Human-readable reason for the action
runId	string	No	Run identifier echoed back on unsolicited slot-release wake-ups (see note)
jobId	string	No	Job identifier echoed back on unsolicited slot-release wake-ups (see note)

The orchestrator sends job.concurrency.ack in two situations: (1) as a direct response to a job.concurrency.report — requestId correlates to the report and action is proceed, wait, or cancel; and (2) as an unsolicited follow-up after the agent received wait. When a slot in the concurrency group is released (by completion, cancellation, or superseding), the orchestrator picks the FIFO-next queued waiter and sends { action: 'proceed' } to the agent still parked on its second waitForConcurrencyAck call. runId and jobId are present in this case so the agent can sanity-check the wake-up matches its job.

Agent metrics push messages

These messages support periodic metrics push from agents to the orchestrator. The orchestrator aggregates agent metrics and exposes them via its Prometheus scrape endpoint.

Authoritative source: packages/engine/src/protocol/messages/orchestrator-agent.ts

agent.metrics

Direction: Agent -> Orchestrator

Periodic metrics push from agent to orchestrator. Each metric includes its type, value, labels, and optional histogram buckets.

Field	Type	Required	Description
type	"agent.metrics"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
agentId	string	Yes	Agent identifier
metrics	MetricEntry[]	Yes	Array of metric data points
timestamp	number	Yes	Unix timestamp (milliseconds)

Each MetricEntry:

Field	Type	Required	Description
name	string	Yes	Metric name
type	enum	Yes	One of: counter, histogram, gauge, upDownCounter
value	number	No	Metric value (for counters and gauges)
labels	Record<string, string>	No	Metric labels
buckets	[{ le: number, count: number }]	No	Histogram bucket boundaries and counts
count	number	No	Histogram observation count
sum	number	No	Histogram observation sum

Authoritative source: packages/engine/src/protocol/messages/orchestrator-agent.ts — agentMetricsSchema

Agent authentication messages

Authentication messages for the agent <-> orchestrator WebSocket connection.

Authoritative source: packages/engine/src/protocol/messages/orchestrator-agent.ts

auth.request (agent)

Auth request sent by agent to orchestrator when connecting.

Field	Type	Required	Description
type	"auth.request"	Yes	Message discriminator
token	string	Yes	Authentication token
protocolVersion	number	Yes	Protocol version (positive integer)

auth.success (agent)

Auth success response sent by orchestrator to agent.

Field	Type	Required	Description
type	"auth.success"	Yes	Message discriminator
connectionId	string	Yes	Assigned connection ID

auth.failure (agent)

Auth failure response sent by orchestrator to agent.

Field	Type	Required	Description
type	"auth.failure"	Yes	Message discriminator
reason	string	Yes	Human-readable failure reason

Authoritative source: packages/engine/src/protocol/messages/orchestrator-agent.ts — agentAuthFailureSchema

Join messages

Join messages carry join protocol v2. A new orchestrator proves it holds a join token without sending the token or its secret. An existing orchestrator answers with its own proof and the configuration bundle sealed to a one-time key of the new orchestrator. A relay sees the token’s routing part, both public keys, both nonces, both proofs and the ciphertext. None of these lets it read or change the bundle. A join.request that carries a token field is refused with join_protocol_v1_removed.

Authoritative source: packages/engine/src/protocol/messages/join.ts

New orchestrator -> existing orchestrator (via Platform relay or direct peer)

join.request

Sent by a new orchestrator to request cluster config from an existing orchestrator.

Field	Type	Required	Description
type	"join.request"	Yes	Message discriminator
messageId	string	No	Correlation ID for Platform relay routing (injected by Platform to match response to correct joiner)
joinProtocol	2	Yes	Join protocol version
routing	string	Yes	The token’s base64url routing part, exactly as it appears in the token
joinerPublicKey	string	Yes	The new orchestrator’s one-time X25519 public key (DER SPKI, base64)
joinerNonce	string	Yes	32 random bytes, base64
joinerProof	string	Yes	HMAC-SHA256 over the request transcript, lowercase hex

Existing orchestrator -> new orchestrator (via Platform relay or direct peer)

join.response

Response from an existing orchestrator with the sealed config bundle, or a refusal.

Field	Type	Required	Description
type	"join.response"	Yes	Message discriminator
messageId	string	No	Correlation ID echoed from join.request for relay routing
success	boolean	Yes	Whether the join was successful
joinProtocol	2	No	Join protocol version (on success)
serverPublicKey	string	No	The existing orchestrator’s one-time X25519 public key (DER SPKI, base64)
serverNonce	string	No	32 random bytes, base64
serverProof	string	No	HMAC-SHA256 over the response transcript, lowercase hex
encryptedBundle	string	No	Base64 AES-256-GCM sealed config bundle (on success)
error	string	No	Error message (on failure)
errorCode	string	No	Why the join was refused (see below). Absent on an internal failure

Join error codes

errorCode	Produced by	Meaning
join_protocol_v1_removed	Platform, orchestrator, route	The request carries the join token (join protocol v1). Upgrade kici-admin
join_protocol_unsupported	None	Not sent at protocol 4: every orchestrator answers join protocol v2. kici-admin join still names the upgrade when it reads this code
invalid_request	Platform, orchestrator, route	The request is malformed
invalid_token	Orchestrator, route	No live join token matches the request’s routing part and proof
token_expired	Orchestrator, route	The matching join token has expired
token_already_used	Orchestrator, route	Another joiner consumed the join token
org_mismatch	Platform	The routing part names an org other than the relaying connection’s own org
no_target	Platform	The target pool holds no orchestrator
relay_timeout	Platform	The target orchestrator did not answer within 30 seconds

“Route” is POST /api/v1/cluster/join on the existing orchestrator, the direct transport of kici-admin join --peer. It answers 200 on success, 400 for invalid_request, 401 for invalid_token, token_expired and token_already_used, 426 with Upgrade: kici-join-v2 for join_protocol_v1_removed, and 500 for an internal failure.

The Browser ↔ Platform protocol (browser-side WebSocket auth, log subscriptions, status fan-out) is documented in the internal docs.

Orchestrator <-> Orchestrator messages (peer-to-peer)

This layer carries cluster coordination messages between orchestrator instances via direct WebSocket connections on /ws/peer. These messages never transit the Platform tier — see Communication Topology.

Authoritative source: packages/engine/src/protocol/messages/peer.ts

Authentication

Peer authentication uses a 4-message ECDH handshake with mutual authentication (mutual-v2). The receiving orchestrator (the server, on /ws/peer) sends peer.hello, the connecting orchestrator (the client) replies with peer.hello.response, then the client sends an encrypted peer.auth.request, and the server replies with an encrypted peer.auth.response. Both messages after the hello exchange use the handshake key K_hs. No credential or join token crosses the wire: each side sends a proof bound to the handshake transcript. The client accepts nothing until the server’s proof verifies. After an acceptance both sides switch to the application key K_app, which needs the shared credential or token as well as K_hs.

peer.hello

Sent by the server when a client connects. Provides the server’s ephemeral ECDH public key, a nonce and the authentication schemes it accepts.

Field	Type	Required	Description
type	"peer.hello"	Yes	Message discriminator
ephemeralPublicKey	string	Yes	Server’s ephemeral X25519 public key (base64, DER SPKI)
nonce	string	Yes	Base64-encoded 32-byte random nonce (HKDF salt)
authSchemes	string[]	No	Schemes the server accepts (mutual-v2). A client refuses a hello without mutual-v2 and sends nothing that depends on a secret

peer.hello.response

Sent by the client in response to peer.hello. Provides the client’s ephemeral ECDH public key. After this exchange, both sides derive K_hs and the transcript hash.

Field	Type	Required	Description
type	"peer.hello.response"	Yes	Message discriminator
ephemeralPublicKey	string	Yes	Client’s ephemeral X25519 public key (base64, DER SPKI)

peer.auth.request

Sent by the client after the ECDH handshake. Encrypted with K_hs. The client proves that it holds its credential (credential mode) or its join token (token mode, first join). The server closes a connection that has not authenticated within 15 seconds. A request that carries proof or token, or no scheme, is refused with the reason Mutual peer authentication required.

Field	Type	Required	Description
type	"peer.auth.request"	Yes	Message discriminator
instanceId	string	Yes	Client’s cluster instance ID
protocolVersion	number	Yes	Protocol version
scheme	enum	Yes	mutual-v2
mode	enum	Yes	credential or token
clientProof	string	Yes	Hex HMAC-SHA256 proof over the handshake transcript
tokenRouting	string	Conditional	The join token’s base64url routing part (token mode only; never the token)
softwareVersion	string	No	Software version of the client (logged)
role	enum	No	Role the client declares: coordinator or worker; the proof covers it

peer.auth.response

Response to a peer authentication request. Encrypted with K_hs.

Field	Type	Required	Description
type	"peer.auth.response"	Yes	Message discriminator
accepted	boolean	Yes	Whether authentication succeeded
instanceId	string	No	Server’s cluster instance ID (present on every acceptance)
serverProof	string	No	Hex proof that the server holds the same credential or token (present on every acceptance)
sessionCredential	string	No	Issued credential for future connections (token mode); the server proof covers it
role	string	No	Role the server grants, from the credential row or the join-token row (present on acceptance)
reason	string	No	Rejection reason (if not accepted)
softwareVersion	string	No	Software version of the server (logged)
agents	PeerAgentSummary[]	No	Server’s connected agent inventory (present when accepted=true)
scalerCapacity	ScalerCapacitySummary[]	No	Server’s scaler capacity (present when accepted=true)
capabilities	PeerCapabilities	No	Server’s feature capabilities (present when accepted=true)

Inventory & consensus

peer.heartbeat

Sent every 30 seconds by each peer. Carries agent inventory, scaler capacity, and Raft consensus state. This is the primary mechanism for routing decisions — the coordinator uses peer heartbeats to determine which peers can handle a job’s label requirements.

Field	Type	Required	Description
type	"peer.heartbeat"	Yes	Message discriminator
instanceId	string	Yes	Sender’s cluster instance ID
term	number	Yes	Current Raft term
leaderId	string or null	Yes	Known Raft leader’s instance ID
draining	boolean	Yes	Whether the sender is gracefully shutting down
agents	PeerAgentSummary[]	Yes	Connected agent inventory
capabilities	PeerCapabilities	Yes	Feature flags (S3 log access, log routing override)
scalerCapacity	ScalerCapacitySummary[]	No	On-demand backend capacity for routing decisions
configVersion	number	No	Shared config version for sync detection
registryVersion	number	No	Registry version for cross-orchestrator registration sync
clusterSettingsVersion	number	No	Shared cluster-settings version for the coordinator-to-worker settings pull
timestamp	number	Yes	Unix timestamp (milliseconds)
hostname	string	No	Machine hostname (os.hostname())
osRelease	string	No	OS kernel release (os.release())
totalMemoryMb	number	No	Total system memory in MiB
memoryUsedMb	number	No	Used memory in MiB
memoryAvailableMb	number	No	Available memory in MiB
cpuCount	number	No	Number of logical CPUs
uptimeSeconds	number	No	System uptime in seconds
nodeVersion	string	No	Node.js version
runningAsUser	string or null	No	Username of the OS user running the orchestrator
runningAsUid	number or null	No	UID of the OS user running the orchestrator
version	string	No	Orchestrator version (e.g., "0.0.1")

PeerAgentSummary:

Field	Type	Required	Description
agentId	string	Yes	Agent identifier
labels	string[]	Yes	Capability labels
activeJobs	number	Yes	Currently running jobs
maxConcurrency	number	Yes	Maximum concurrent jobs
platform	string	Yes	OS platform (e.g., linux)
arch	string	Yes	CPU architecture (e.g., x64, arm64)
mandatoryLabels	string[]	Yes	Taint gate inherited from the spawning scaler. A static agent sends [], which is “no gate”
scalerName	string or null	No	Backend that spawned this agent, or null for a static agent. Groups the peer’s agents in diagnostics

PeerCapabilities:

Field	Type	Required	Description
s3LogAccess	boolean	Yes	Whether this peer has direct S3 log access
logRoutingOverride	enum	No	One of: direct, coordinator (overrides default routing)

ScalerCapacitySummary:

Field	Type	Required	Description
name	string	No	Scaler backend name (e.g., worker-bare-metal)
type	string	No	Scaler backend type (e.g., bare-metal, container)
labelSets	string[][]	Yes	Label sets this backend provisions
maxAgents	number	Yes	Maximum agents for this backend
activeCount	number	Yes	Current active agent count
spawnsOnLocalHost	boolean	No	Whether this backend spawns its agents on the peer’s own host
labelSetMandatoryLabels	string[][]	Yes	Taint gate per label set, index-aligned with labelSets; an empty entry is “no gate”. An entry whose length does not match routes nothing

The object is strict: a frame carrying any other field fails to parse.

raft.vote.request

Raft leader election vote request. Sent by candidates during elections.

Field	Type	Required	Description
type	"raft.vote.request"	Yes	Message discriminator
term	number	Yes	Candidate’s current term
candidateId	string	Yes	Candidate’s instance ID
lastLogIndex	number	Yes	Candidate’s last log index
lastLogTerm	number	Yes	Term of candidate’s last log entry

raft.vote.response

Response to a Raft vote request.

Field	Type	Required	Description
type	"raft.vote.response"	Yes	Message discriminator
term	number	Yes	Voter’s current term
voteGranted	boolean	Yes	Whether the vote was granted
voterId	string	Yes	Voter’s instance ID

raft.append.entries

Raft leader heartbeat (no log entries — KiCI uses Raft for leader election only, not log replication).

Field	Type	Required	Description
type	"raft.append.entries"	Yes	Message discriminator
term	number	Yes	Leader’s current term
leaderId	string	Yes	Leader’s instance ID

Job rerouting

job.reroute

Sent by the coordinator to a peer when no local agent can handle a job. Contains the full resolved job configuration so the peer can dispatch without re-resolving.

Field	Type	Required	Description
type	"job.reroute"	Yes	Message discriminator
messageId	string	Yes	Unique message ID for ACK correlation
jobId	string	Yes	Pre-allocated job ID. The sending coordinator allocates it before reroute so its execution_runs / execution_jobs rows reference the same id the receiving peer will dispatch under.
runId	string	Yes	Execution run identifier
deliveryId	string	Yes	Original webhook delivery ID
routingKey	string	Yes	Provider routing key
event, action	string, string or null	Yes	Webhook event type and action
payload	Record<string, unknown>	Yes	Full webhook payload
jobName	string	Yes	Job to execute
workflowName	string	Yes	Workflow containing the job
runsOnLabels	string[][]	Yes	Label sets the job requires
excludeLabels	string[]	No	Labels that the dispatched agent must NOT have
runsOnPatterns	LabelMatcher[]	No	Glob/regex include matchers the receiving peer’s agent labels must satisfy, applied on top of the exact runsOnLabels prefilter. Absent reads as []; a pure-pattern job carries its whole selector here
excludePatterns	LabelMatcher[]	No	Glob/regex matchers that disqualify a candidate agent. Absent reads as []
triedConnections	string[]	Yes	Instance IDs already tried (loop prevention)
maxHops	number	Yes	Maximum allowed hops (default: 3)
spawnRetry	{ maxAttempts: number, backoffMs: number }	Yes	Spawn-retry budget the receiving worker applies to the job: agent spawns it attempts (>= 1) and the wait after a failed one (ms, >= 0)
coordinatorId	string	Yes	Instance ID of the run coordinator
jobConfig	Record<string, unknown>	No	Resolved job config (steps, rules, matrix, etc.)
repoUrl	string	No	Repository clone URL
ref	string	No	Git ref (branch name)
sha	string	No	Commit SHA
provider	string	No	Provider type (e.g., github)
providerContext	Record<string, unknown>	No	Provider-specific context (e.g., installationId)
sourceTarUrl	string	No	Pre-signed .kici/ source tarball download URL (cache hit)
sourceTarDigest	string	No	SHA-256 of the source tarball’s own bytes, for integrity verification
depsUrl	string	No	Pre-signed dependency tarball URL (cache hit)
depsHash	string	No	SHA-256 of the dependency tarball bytes
cloneToken	string	No	Pre-resolved clone token for workers without provider credentials
workflowCloneToken	string	No	Pre-resolved clone token for the workflow repository of an organization-wide (global) job
encryptedSecrets	string	No	Encrypted secrets envelope (AES-256-GCM with session key)
encryptedNamespacedSecrets	string	No	Encrypted namespaced secrets envelope
requestId	string	No	Trace ID for distributed tracing
traceId	string	No	Additional trace context

LabelMatcher: { kind: "exact", value: string } or { kind: "regex", source: string, flags: string }. Globs are converted to regex at compile time, so the wire only ever carries these two forms.

job.reroute.ack

Acknowledgment of a reroute request. The coordinator tries the next peer if rejected.

Field	Type	Required	Description
type	"job.reroute.ack"	Yes	Message discriminator
messageId	string	Yes	ID of the job.reroute being acknowledged
accepted	boolean	Yes	Whether the peer accepted the job
reason	string	No	Rejection reason (if not accepted)

Progress & cancel

job.progress

Sent by the worker orchestrator back to the coordinator as the agent reports job-level and step-level state changes. The coordinator uses this to drive its run-level status tracking, update GitHub check runs, and track per-step progress.

The kind discriminator decides which ExecutionTracker call the receiver makes:

• kind: "job" → onJobStatus(runId, jobId, state, ...) — drives run transitions (running → success/failed/…). stepIndex/stepName are unused for this kind.
• kind: "step" → onStepStatus(runId, jobId, stepIndex, stepName, state, ...) — persists execution_steps rows.

Field	Type	Required	Description
type	"job.progress"	Yes	Message discriminator
kind	"job" | "step"	Yes	Whether this update is a job-level or step-level transition
runId	string	Yes	Execution run identifier
jobId	string	Yes	Job identifier (matches the pre-allocated jobId from job.reroute)
jobName	string	Yes	Human-readable job name
stepIndex	number	Yes	Zero-based step index (used only when kind="step")
stepName	string	Yes	Human-readable step name (used only when kind="step")
state	enum	Yes	Full ExecutionJobStatus for kind="job"; the step-state subset (running/success/failed/skipped) for kind="step"
timestamp	number	Yes	Unix timestamp (milliseconds)
data	Record<string, unknown>	No	Optional state-specific data

job.progress.ack

Sent by the coordinator back to the worker once it has applied a terminal job-level job.progress (kind: "job") to its run/job rows. The worker uses it to prune the matching record from its durable terminal-status outbox, so a status that has been durably observed by the coordinator is delivered exactly once. (runId, jobId) is the dedup key; state is carried for debuggability.

Field	Type	Required	Description
type	"job.progress.ack"	Yes	Message discriminator
runId	string	Yes	Execution run identifier
jobId	string	Yes	Job identifier (matches the acknowledged job.progress)
state	enum	Yes	The applied ExecutionJobStatus

peer.job.cancel

Sent by a coordinator to the coordinator or worker that holds the job’s agent. Used for fail-fast propagation and user-initiated cancellation.

Field	Type	Required	Description
type	"peer.job.cancel"	Yes	Message discriminator
runId	string	Yes	Execution run identifier
jobId	string	No	Specific job to cancel; omit to cancel every job of the run the receiver tracks
reason	string	Yes	Human-readable cancellation reason
force	boolean	No	When true, force-cancel immediately without waiting for hooks

Graceful shutdown

peer.leaving

Graceful shutdown announcement. Peers remove the sender from their registry immediately upon receiving this message.

Field	Type	Required	Description
type	"peer.leaving"	Yes	Message discriminator
instanceId	string	Yes	Instance ID of the leaving peer
term	number	Yes	Current Raft term for leader identification

Log and cache relay (coordinator-worker topology)

peer.log.chunk

Log chunk relay from worker to coordinator. Batched log lines from agent execution, forwarded when the worker does not have direct S3 log access.

Field	Type	Required	Description
type	"peer.log.chunk"	Yes	Message discriminator
runId	string	Yes	Execution run ID
jobId	string	Yes	Job ID within the run
stepIndex	number	Yes	Zero-based step index
lines	LogLineEntry[]	Yes	Array of log line entries

Each LogLineEntry:

Field	Type	Required	Description
text	string	Yes	Log line text
timestamp	number	Yes	Unix timestamp (milliseconds)
stream	enum	Yes	One of: stdout, stderr

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerLogChunkSchema

peer.cache.upload.request

Cache upload request from worker to coordinator. Worker agent needs an upload URL for cache storage.

Field	Type	Required	Description
type	"peer.cache.upload.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
runId	string	Yes	Execution run ID
jobId	string	Yes	Job ID within the run
cacheType	enum	Yes	One of: source, deps
hash	string	Yes	Content hash for cache keying
sizeBytes	number	Yes	Size of the artifact in bytes

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerCacheUploadRequestSchema

peer.cache.upload.response

Cache upload response from coordinator to worker. Pre-signed upload URL for the worker to upload directly to object storage.

Field	Type	Required	Description
type	"peer.cache.upload.response"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
runId	string	Yes	Execution run ID
jobId	string	Yes	Job ID within the run
uploadUrl	string	Yes	Pre-signed URL for direct S3 upload

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerCacheUploadResponseSchema

Fleet log collection across peers

When an operator collects a debug bundle that spans the cluster, a coordinator fans a collection request down the peer mesh and each peer assembles and streams back the ZIP for its own subtree.

peer.logs.collect.request

Ask a peer to assemble and stream back the log bundle for its downstream subtree.

Field	Type	Required	Description
type	"peer.logs.collect.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
logWindowHours	number	Yes	How many hours of logs to gather
includeCoordinatorMesh	boolean	Yes	Loop guard: false on every downstream request so the coordinator mesh never echoes
selection	FleetSelection	Yes	Which downstream nodes to gather

Each FleetSelection:

Field	Type	Required	Description
all	boolean	Yes	Gather every downstream node; ignores the id lists below
agentIds	string[]	No	Specific agent IDs to gather (defaults to empty)
workerInstanceIds	string[]	No	Specific worker instance IDs to gather (defaults to empty)

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerLogsCollectRequestSchema

peer.logs.collect.chunk

One base64-encoded frame of a peer’s subtree ZIP, streamed back to the requesting coordinator.

Field	Type	Required	Description
type	"peer.logs.collect.chunk"	Yes	Message discriminator
messageId	string	Yes	Unique message ID (matches the request)
seq	number	Yes	Zero-based, non-negative frame sequence number
isLast	boolean	Yes	Whether this is the final frame
dataB64	string	Yes	Base64-encoded ZIP frame

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerLogsCollectChunkSchema

peer.logs.collect.error

Sent when a peer fails to assemble or stream its subtree bundle.

Field	Type	Required	Description
type	"peer.logs.collect.error"	Yes	Message discriminator
messageId	string	Yes	Unique message ID (matches the request)
message	string	Yes	Failure reason

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerLogsCollectErrorSchema

Config reload (per-instance targeting)

peer.config.reload

Config reload request forwarded from one orchestrator to a specific peer when an operator calls POST /admin/config/reload with a target parameter. The receiving peer executes a local reload and replies with peer.config.reload.response.

Field	Type	Required	Description
type	"peer.config.reload"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
drain	boolean	No	Whether to drain in-flight work before reloading

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerConfigReloadSchema

peer.config.reload.response

Response from the target peer carrying the reload result fields.

Field	Type	Required	Description
type	"peer.config.reload.response"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
success	boolean	Yes	Whether the reload succeeded
version	number	No	New config version after reload
errors	string[]	No	Error messages if reload failed
restartRequired	string[]	No	Config fields that require a restart to take effect
fieldsChanged	string[]	No	Config fields that were changed
scaler	object	No	The target’s scaler config reload: outcome, and plan or errors (see peer.scaler.reload.response)

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerConfigReloadResponseSchema, scalerReloadAnswerSchema

peer.clusterSettings.request

Cluster-settings pull request: a DB-less worker asks its first connected coordinator for the current worker-settings snapshot after a coordinator advertised a clusterSettingsVersion (on peer.heartbeat) ahead of its own. Every coordinator reads the same cluster_settings row, so any coordinator answers. The coordinator replies with peer.clusterSettings.response.

Field	Type	Required	Description
type	"peer.clusterSettings.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerClusterSettingsRequestSchema

peer.clusterSettings.response

Cluster-settings pull response: the coordinator resolves the worker-relevant settings snapshot from its cluster settings and replies with it plus the current version. The snapshot is a typed, concrete object (not an open key/value bus) — each worker-consumed cluster knob is a field on it.

Field	Type	Required	Description
type	"peer.clusterSettings.response"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
version	number	Yes	Current cluster-settings version
settings	WorkerClusterSettings	Yes	Worker-settings snapshot (e.g., agentTokenTtlMs)

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerClusterSettingsResponseSchema, workerClusterSettingsSchema

Scaler orphans (per-instance targeting)

peer.scaler.orphans.request

Request forwarded from a coordinator to the node an operator targeted with kici-admin scaler orphans --target. The node answers from its own host and its own tracking with peer.scaler.orphans.response.

Field	Type	Required	Description
type	"peer.scaler.orphans.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
action	"list" | "stop"	Yes	List the live Firecracker VMs, or stop some of them
vmIds	string[]	No	stop only: the VM IDs the operator approved

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerScalerOrphansRequestSchema

peer.scaler.orphans.response

Response from the target node. vms answers a list, and results answers a stop. When ok is false, the response carries the node’s error instead.

Field	Type	Required	Description
type	"peer.scaler.orphans.response"	Yes	Message discriminator
messageId	string	Yes	Matches the request
ok	boolean	Yes	Whether the node answered the request
error	string	No	Why the node could not answer
firecrackerScalers	string[]	No	The node’s Firecracker scaler names; empty when it runs none
vms	object[]	No	Each live VM: vmId, scaler, pid, startedAt, ageSeconds, chrootDir, status, trackedBy, reason
results	object[]	No	Each stop result: vmId, outcome, pid, detail

A VM status is orphaned, unverified or tracked. A stop outcome is stopped, tracked, unverified, not-live, not-found or error. Only stopped signalled a process.

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerScalerOrphansResponseSchema

Scaler reload fan-out

peer.scaler.reload.request

The orchestrator that received kici-admin scaler reload (without --single) sends this message to each connected peer, coordinators and workers alike, so each one re-reads its own scaler config. A peer accepts it from coordinators only: the incoming peer connection checks that the sender’s credential is a coordinator’s.

Field	Type	Required	Description
type	"peer.scaler.reload.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerScalerReloadRequestSchema

peer.scaler.reload.response

What the peer’s reload did. outcome is applied, rejected, not-configured or unreachable. A rejected reload applied nothing.

Field	Type	Required	Description
type	"peer.scaler.reload.response"	Yes	Message discriminator
messageId	string	Yes	Matches the request
outcome	string	Yes	What the reload did on this peer
plan	object	No	applied only: scaler names per bucket (added, updated, unchanged, retired, resurrected) and changed global limits
errors	string[]	No	rejected only: why the file was refused
detail	string	No	Why the peer could not reload (for example, a request from a peer that is not a coordinator)

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerScalerReloadResponseSchema, scalerReloadPlanSchema

Peer forget fan-out

peer.forget.request

A coordinator that forgot a departed peer (kici-admin peer forget) sends this message to each connected sibling coordinator, so each one drops the peer from its own live peer registry. A peer accepts it from coordinators only.

Field	Type	Required	Description
type	"peer.forget.request"	Yes	Message discriminator
messageId	string	Yes	Unique message ID
instanceId	string	Yes	The departed peer’s instance ID
acknowledgeBackstop	boolean	No	The operator acknowledged that the forget may switch the event-provision backstop back on

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerForgetRequestSchema

peer.forget.response

What the sibling did with the request. outcome is forgotten, not-found, connected, recent, acknowledgement-required or error. Every outcome except forgotten keeps the peer.

Field	Type	Required	Description
type	"peer.forget.response"	Yes	Message discriminator
messageId	string	Yes	Matches the request
outcome	string	Yes	What the forget did on this coordinator
detail	string	Yes	The reason, readable by an operator

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerForgetResponseSchema

Agent-token revoke fan-out

peer.agent-token.revoke

Broadcast to every peer when an agent token is revoked so each peer can close its own in-flight agent WS connections authenticated by that token. The originating peer kicks its local connections first (via the admin revoke route), then fans this message out over the encrypted peer mesh.

Field	Type	Required	Description
type	"peer.agent-token.revoke"	Yes	Message discriminator
tokenId	string	Yes	The agent_tokens.id whose in-flight agent WS must be kicked
senderInstanceId	string	Yes	Originating peer’s instance ID (for cross-cluster log correlation)

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerAgentTokenRevokeSchema

Scaler provisioning events

scaler.event

Forwarded by a worker to the coordinator that owns the run when the worker’s scaler emits a provisioning event correlated to a queued job (e.g. a failed agent spawn). Workers have no database, so they cannot persist provisioning failures themselves — the coordinator’s ExecutionTracker writes the event to the provisioning log and the dispatch queue’s last-error column.

Field	Type	Required	Description
type	"scaler.event"	Yes	Message discriminator
runId	string	Yes	Execution run ID
jobId	string	Yes	Job ID within the run
agentId	string	Yes	The scaler-managed agent ID the event is about
eventType	enum	Yes	Scaler event type (a ScalerEventType enum member)
detail	string	Yes	Human-readable detail, including any captured spawn stderr tail
timestampMs	number	Yes	Event timestamp in epoch milliseconds
final	boolean	No	Retry verdict on a scaler.failed for a job under the worker’s spawn-retry budget: false while attempts remain, true on the last one. Absent on a repeated report of one failed spawn and on every other relay

Authoritative source: packages/engine/src/protocol/messages/peer.ts — peerScalerEventSchema

See Multi-Orchestrator Architecture for rerouting protocol, loop prevention, and failure modes.

Test-relay control messages

These messages implement the Platform-routed kici run remote flow. On this path the CLI does not connect to the orchestrator: it authenticates to the Platform. The Platform proxies each control message to the customer’s orchestrator over the dashboard-direction WebSocket and relays the response back to the CLI. Every request carries an actor principal (so the orchestrator can authorize the operation) and a requestId (so the Platform can correlate the matching response).

Authoritative source: packages/engine/src/protocol/messages/dashboard.ts

The orchestrator also serves the same handlers over HTTP at /api/v1/test/* for a direct run, authenticated with an orchestrator admin token. Those routes accept the request bodies below minus type, requestId, actor and routingKey, which the orchestrator sets itself.

The control plane is five request/response pairs. Requests travel Platform -> Orchestrator (members of dashboardPlatformToOrchSchema); responses travel Orchestrator -> Platform (members of dashboardOrchToPlatformSchema). On every response the optional error string carries a failure reason when the operation could not be served.

Platform -> Orchestrator

test.relay.uploads.init

Request a presigned URL for uploading the working-tree overlay tarball.

Field	Type	Required	Description
type	"test.relay.uploads.init"	Yes	Message discriminator
requestId	string	Yes	Correlation ID for the response
actor	ActorPrincipal	Yes	Authenticated principal initiating the run
routingKey	string	Yes	Provider routing key
sha	string	No	Content hash of the overlay being uploaded
fileCount	number	No	Number of files in the overlay
compressedSize	number	No	Compressed overlay size in bytes

test.relay.trigger

Trigger a remote test run using a fixture payload (mirrors the orchestrator’s TestTriggerInput).

Field	Type	Required	Description
type	"test.relay.trigger"	Yes	Message discriminator
requestId	string	Yes	Correlation ID for the response
actor	ActorPrincipal	Yes	Authenticated principal initiating the run
routingKey	string	Yes	Provider routing key
fixtureId	string	Yes	Fixture identifier
event	TestEvent	Yes	Simulated event (type, action, branch, payload)
workflowName	string	No	Specific workflow to test (all if omitted)
uploadId	string	No	Upload ID returned by test.relay.uploads.init.response
cliPublicKey	string	No	CLI ephemeral X25519 public key used to encrypt the overlay tarball
inlineLockFile	string	No	Lock file sent inline instead of via upload
fullRepo	boolean	No	Whether the upload is the full working tree (not an overlay)
checkMode	enum	No	Run mode for idempotent steps, relayed onto the dispatch event. Omitted means apply
target	HostTargetSelector	No	Host narrowing from kici run remote --target, intersected with each runsOnAll job’s matched roster. Omitted for webhook runs
dispatchInputs	Record<string, string>	No	Raw kici run remote --input KEY=VALUE pairs, relayed verbatim. The orchestrator validates, coerces and defaults them against the matched workflow’s lock descriptor
secrets	Record<string, string>	No	Plaintext secrets for the run
encryptedSecrets	string	No	Encrypted secrets blob
encryptedSecretsKey	string	No	Key wrapping the encrypted secrets blob
acceptsMultipleRuns	boolean	No	The caller follows every run in the response’s runs, so a fixture that matches several workflows gets one run per workflow. Without it the orchestrator answers as for a caller that follows only runId

TestEvent: { type: string, action?: string, targetBranch: string, sourceBranch?: string, payload: Record<string, unknown>, changedFiles?: string[] }

test.relay.run.status

Request a snapshot of a run’s status.

Field	Type	Required	Description
type	"test.relay.run.status"	Yes	Message discriminator
requestId	string	Yes	Correlation ID for the response
actor	ActorPrincipal	Yes	Authenticated principal
runId	string	Yes	Execution run ID

test.relay.run.logs

Request the next chunk of a run’s logs from a cursor.

Field	Type	Required	Description
type	"test.relay.run.logs"	Yes	Message discriminator
requestId	string	Yes	Correlation ID for the response
actor	ActorPrincipal	Yes	Authenticated principal
runId	string	Yes	Execution run ID
cursor	number	Yes	Lines of the sorted stream already delivered; read when logCursor is absent
logCursor	string	No	Opaque per-file cursor: a prior nextLogCursor, or the empty string to start. When present, the orchestrator reads from it and ignores cursor

test.relay.cancel

Request cancellation of a run.

Field	Type	Required	Description
type	"test.relay.cancel"	Yes	Message discriminator
requestId	string	Yes	Correlation ID for the response
actor	ActorPrincipal	Yes	Authenticated principal
runId	string	No	Run ID to cancel
branch	string	No	Cancel the active run for this branch

Orchestrator -> Platform

test.relay.uploads.init.response

Presigned upload URL plus the ephemeral encryption key the CLI uses to encrypt the overlay.

Field	Type	Required	Description
type	"test.relay.uploads.init.response"	Yes	Message discriminator
requestId	string	Yes	ID of the request being responded to
uploadId	string	No	Upload record ID
signedUrl	string	No	Presigned PUT URL for the overlay tarball
publicKey	string	No	Orchestrator public key for overlay encryption
expiresIn	number	No	URL validity window in seconds
error	string	No	Failure reason (when the URL could not be minted)

test.relay.trigger.response

Acknowledge a triggered run.

Field	Type	Required	Description
type	"test.relay.trigger.response"	Yes	Message discriminator
requestId	string	Yes	ID of the request being responded to
runId	string	No	Deprecated: read runs. The first run of runs, or the one run when the request did not set acceptsMultipleRuns
runs	TriggeredRun[]	No	Every run the trigger created, one per matched workflow. Absent from an orchestrator that predates the field; read runId then
status	enum	No	One of: accepted, rejected
reason	string	No	Rejection reason (if rejected)
jobIds	string[]	No	IDs of the jobs created for the run
warnings	string[]	No	User-visible warnings on acceptance (e.g. a skipped non-test bound context)
error	string	No	Failure reason

TriggeredRun: { runId: string, workflowName: string, jobIds?: string[] }

test.relay.run.status.response

A run-status snapshot.

Field	Type	Required	Description
type	"test.relay.run.status.response"	Yes	Message discriminator
requestId	string	Yes	ID of the request being responded to
runId	string	No	Execution run ID
status	string	No	Run status
jobs	array	No	Per-job status: { jobId, jobName, status, exitCode?, errorMessage?, durationMs? }. durationMs is the job’s duration in milliseconds; when it is absent, the CLI shows the duration as unknown
failureReason	string | null	No	Why the run failed, when it failed before or outside any job (an init failure records no job). Absent from an orchestrator that predates the field
done	boolean	No	Whether the run has reached a terminal state
error	string	No	Failure reason

test.relay.run.logs.response

The next log chunk and the cursors to read on from.

Field	Type	Required	Description
type	"test.relay.run.logs.response"	Yes	Message discriminator
requestId	string	Yes	ID of the request being responded to
lines	string[]	No	Log output lines
nextCursor	number	No	Cursor for the next test.relay.run.logs request
nextLogCursor	string	No	Opaque per-file cursor for the next request; present when the request sent logCursor
lineJobNames	string[]	No	The job that wrote each line, index for index with lines. Present when the request sent logCursor; absent from an orchestrator that predates the field
done	boolean	No	Whether the log stream is exhausted
error	string	No	Failure reason

test.relay.cancel.response

Acknowledge a cancellation.

Field	Type	Required	Description
type	"test.relay.cancel.response"	Yes	Message discriminator
requestId	string	Yes	ID of the request being responded to
cancelled	boolean	No	Whether the run was cancelled
error	string	No	Failure reason

WebSocket close codes

KiCI defines custom close codes in the 4000-4999 range (reserved for application use by RFC 6455) alongside standard codes.

Code	Constant	Meaning	Sent by
1000	—	Normal closure. Sent by the connecting side when it disconnects on purpose — an agent or orchestrator shutting down its uplink, or any client (agent, orchestrator uplink, or peer) abandoning a handshake that failed	Client
1001	WS_CLOSE_GOING_AWAY	Server shutdown or browser navigating away. The orchestrator also sends it to an agent that reaches the ownership-violation threshold (5 frames for jobs it does not own within 60 seconds)	Server
4001	WS_CLOSE_UNAUTHORIZED	Client failed authentication. Also sent when the Platform times out an unauthenticated connection, when a client’s API key is revoked mid-session, on an operator force-disconnect, and when a dashboard session loses its org membership or exceeds its connection limit. On the orchestrator peer channel it closes a rate-limited, unauthenticated, or role-mismatched peer	Server
4002	WS_CLOSE_AUTH_TIMEOUT	Auth timeout expired. The orchestrator also sends it to an agent that authenticates but does not send agent.register in time. On the orchestrator peer channel it closes a peer that does not finish authenticating in time	Server
4003	WS_CLOSE_INVALID_MESSAGE	Invalid or unparseable message received. The orchestrator also sends it to an agent whose first message is not agent.register, that exceeds the message rate limit, or that registers an agent ID already bound to a different token. The Platform sends it to an orchestrator whose frame is not JSON or has no type, or whose auth message is invalid. It also closes an orchestrator that exceeds the rate limit, sends an invalid cluster_name, or sends five invalid frames within 60 seconds. The Platform dashboard channel sends it when a browser’s first message is not auth.request. The peer channel sends it for a malformed handshake frame	Server
4004	WS_CLOSE_HEARTBEAT_TIMEOUT	Heartbeat timed out (180 seconds)	Server
4005	WS_CLOSE_PROTOCOL_ERROR	Protocol-level error: an unsupported protocol version, or, on the Platform channel, an auth message on an authenticated connection, or any other message before authentication. On the orchestrator peer channel it closes a peer whose handshake breaks the protocol or whose authentication scheme lacks mutual authentication	Server
4006	WS_CLOSE_INTERNAL_ERROR	Unexpected internal server error	Server
4010	WS_CLOSE_AGENT_AUTH_FAILED	Agent token authentication failed. The orchestrator also sends it to every agent connected under a token that is revoked or whose time-to-live expires. At registration, it also sends it when the agent’s labels exceed the token’s label scope, when an ephemeral token is bound to a different agent ID, or when a non-root agent presents a privileged-root token. On the Platform-orchestrator channel the same code is also sent as a safety fallback when every source in a source.register is rejected for missing S3 log storage in a multi-orchestrator pool	Server
4011	WS_CLOSE_CLUSTER_NAME_CONFLICT	Reserved constant; no Platform code path emits this today. Platform accepts N connected orchestrators per (org_id, cluster_name) and the dashboard listing dedupes by cluster name	—
4020	WS_CLOSE_PLAN_LIMIT	Organization has reached its plan limit. The Platform sends it when it refuses a new orchestrator connection over the org’s limit, or closes an excess one. A coordinator sends it when it refuses a worker join or drains a connected worker because the org is over its orchestrator ceiling (coordinators and workers combined)	Server
4030	WS_CLOSE_REBALANCE	Connection rebalancing: a newly-joined Platform instance asked peers holding excess connections to shed some; the orchestrator reconnects immediately and load redistributes evenly	Server
4031	WS_CLOSE_DISPATCH_ACK_TIMEOUT	A dispatched job went unacknowledged past its deadline; the orchestrator requeues the job and disconnects the unresponsive agent	Server
4032	WS_CLOSE_SUPERSEDED_BY_RECONNECT	The same agent registered on a newer socket, so the older one is a ghost. The orchestrator retires it at registration time, before the OS or the WebSocket ping reaps it, because a half-open socket carries the same agent id and its later close would tear down the live registration. The agent is already connected on the newer socket and does not reconnect	Server

Authoritative source: packages/engine/src/ws/close-codes.ts

Validation

All protocol messages are validated at runtime using Zod discriminated unions. Each WebSocket layer has direction-specific schemas that prevent parsing messages from the wrong direction.

Orchestrator-Agent layer:

• Incoming (Orchestrator receives): agentToOrchestratorMessageSchema — parses agent.register, agent.status, job.status, job.reject, job.ack, log.chunk, step.status, job.heartbeat, agent.log, job.concurrency.report, config.ack, cache.upload.request, cache.upload.complete, cache.user.restore.request, cache.user.save.request, cache.user.save.complete, provenance.upload.request, provenance.upload.complete, provenance.upload.defer, artifacts.upload.request, artifacts.upload.complete, artifacts.download.request, event.emit, scaler.claim-credentials, agent.api.request, agent.metrics, auth.request, fleet.bundle.chunk, fleet.bundle.error, step.approval-request
• Outgoing (Orchestrator sends): orchestratorToAgentMessageSchema — validates job.dispatch, job.cancel, register.ack, job.concurrency.ack, cache.upload.response, cache.user.restore.response, cache.user.save.response, provenance.upload.response, artifacts.upload.response, artifacts.upload.complete.ack, artifacts.download.response, event.emit.response, scaler.claim-credentials.response, agent.api.response, auth.success, auth.failure, fleet.logs.request, step.approval-resolved, rate.limit.warning

Peer-to-peer layer:

• Bidirectional: peerToPeerMessageSchema / peerFromPeerMessageSchema — parses peer.hello, peer.hello.response, peer.auth.request, peer.auth.response, peer.heartbeat, job.reroute, job.reroute.ack, job.progress, job.progress.ack, peer.job.cancel, raft.vote.request, raft.vote.response, raft.append.entries, peer.log.chunk, peer.cache.upload.request, peer.cache.upload.response, peer.config.reload, peer.config.reload.response, peer.scaler.orphans.request, peer.scaler.orphans.response, peer.scaler.reload.request, peer.scaler.reload.response, peer.forget.request, peer.forget.response, peer.clusterSettings.request, peer.clusterSettings.response, peer.logs.collect.request, peer.logs.collect.chunk, peer.logs.collect.error, peer.leaving, peer.agent-token.revoke, scaler.event

The upstream layers (orchestrator↔KiCI and browser↔KiCI) have their own discriminated unions.

Usage example:

import { agentToOrchestratorMessageSchema } from '@kici-dev/engine';

// Parse and validate an incoming message
const message = agentToOrchestratorMessageSchema.parse(JSON.parse(rawData));

// TypeScript narrows the type based on the 'type' discriminator
switch (message.type) {
  case 'agent.register':
    handleRegister(message.agentId, message.labels);
    break;
  case 'job.status':
    handleStatus(message.runId, message.jobId, message.status);
    break;
}

Request tracing model

KiCI propagates a requestId (UUIDv4) through the entire webhook processing pipeline for end-to-end observability. The trace ID enables correlating all log lines, database operations, and protocol messages belonging to a single webhook event.

requestId lifecycle

1. Origin: Generated at webhook ingestion — either by KiCI (for WS-relayed webhooks) or by the orchestrator’s direct HTTP endpoint (for independent/hybrid mode)
2. Pipeline propagation: Carried through AsyncLocalStorage (ALS) during synchronous webhook processing: webhook.relay -> processWebhook -> trigger matching -> execution start -> job dispatch
3. Queue persistence: When no agent is immediately available, the requestId is persisted in the dispatch_queue PostgreSQL table alongside the job payload
4. Queue restoration: When a queued job is drained (agent becomes available), the requestId is read from the database and restored into a new ALS scope via requestContext.run()
5. Agent dispatch: Included in the job.dispatch message so the agent can log it during execution
6. Completion callbacks: Stored in the ExecutionTracker’s in-memory RunState at execution start, then passed back to onExecutionComplete and onStepStatusForward callbacks. The callback handlers in server.ts and standalone.ts restore ALS context via requestContext.run() so that check run updates and upstream forwarding are logged with the correct trace ID.

app.service field

The app.service field identifies which KiCI tier produced a log line (values: platform, orchestrator, agent). Set process-wide at startup via setServiceName() from @kici-dev/shared. This field is independent of container naming and works in all deployment models (containerized, bare-metal, Firecracker). For forwarded agent logs flowing through orchestrator stdout, the service field is preserved from the agent’s original JSON output.

Unsupported request types

A dashboard request reaches the orchestrator without a pre-flight check: every orchestrator at protocol 4 understands every dashboard.* request type the Platform sends. When a request fails schema validation, the orchestrator still distinguishes a request type it has never heard of (unsupported_request_type, “upgrade the orchestrator”) from a known type with a malformed body (invalid_payload, a genuine client error). The error response frame carries the structured code and the orchestrator version.

See also