Architecture overview

KiCI uses a three-tier relay model that separates webhook routing from code execution. Customer code never leaves customer infrastructure — the Platform tier handles only webhook verification and routing, while the orchestrator and agent tiers run on customer-managed servers.

Three-tier relay model

The system is organized into three deployment tiers connected by WebSocket channels:

Why three tiers? Trust boundaries. The Platform relay never sees customer code or webhook secrets — it only forwards payloads, and the orchestrator verifies their signatures. The orchestrator matches triggers against the lock file without cloning repositories. Only the agent, running on customer infrastructure, clones code and executes steps.

This model also supports pointing webhooks directly at the orchestrator, bypassing the Platform relay. In hybrid mode the orchestrator keeps its Platform connection for the dashboard and telemetry while GitHub delivers events straight to the orchestrator’s ingress — so a Platform outage never drops a build trigger. Observed mode drops the relay leg entirely: webhooks reach only the orchestrator’s own ingress (no payload ever transits KiCI) while the Platform connection stays up for the hosted dashboard, and the orchestrator’s sources register as observe-only — recorded and dashboard-visible, but excluded from every relay-candidate lookup. In a fully independent deployment the orchestrator and agent run on customer infrastructure with no Platform at all, receiving webhooks directly. For exactly which capabilities the hosted Platform provides in each case, see What requires the hosted Platform.

Component responsibilities

Platform

The Platform is KiCI’s hosted, multi-tenant control plane. It provides the hosted dashboard (run listing, run detail, live log streaming, settings), identity and authentication (OIDC, personal access tokens, API keys, JWTs), multi-tenant organization / team / role-based access management, billing, and webhook ingestion. Webhook ingestion relays payloads to the correct orchestrator over WebSocket. The orchestrator verifies their signatures (HMAC-SHA256, timing-safe) against secrets the Platform never holds. It aggregates execution telemetry and status forwarded by orchestrators, registers sources, and matchmakes peers for clustering.

The Platform never processes, stores, or executes customer code, and never sees customer secrets. It routes webhook payloads and aggregates execution status; the code itself only ever lives on the customer’s orchestrator and agent tiers. In the execution path the Platform is deliberately thin — it does not run jobs — but functionally it is a full platform, not merely a relay. The hosted Platform is EU-sovereign.

Orchestrator (`@kici-dev/orchestrator`)

The orchestrator is the execution brain. It decides what to run and dispatches work to agents.

Trigger matching — Evaluates lock file triggers against webhook payloads to determine which jobs to run. Uses glob-based branch, path, and event matching.
Lock file caching — Fetches kici.lock.json via the configured source’s fetcher (GitHub API, universal-git clone for generic webhook sources backed by a git URL, or the local filesystem for file:// sources). An LRU cache wraps the per-provider fetcher, keyed by {provider}:{repo}:{ref} so cross-provider fallback resolutions stay isolated. The local-filesystem fetcher opts out of the cache: a file on disk can change under a fixed ref, so every lookup reads the current file.
Agent registry — Tracks connected agents with label-based routing for job dispatch.
Job queue — PostgreSQL-backed FIFO queue for reliable dispatch.
Webhook pipeline — Dedup, event mapping, lock file fetch, trigger matching, and job dispatch in a single pipeline.
Multi-orchestrator clustering — Optional peer-to-peer coordination via direct WebSocket connections. Enables cross-architecture job routing (e.g., x64 coordinator reroutes arm64 jobs to a peer), high availability, and dedicated coordinator topologies. Uses Raft consensus for leader election (orphan recovery). See Multi-orchestrator architecture.
Auto-scaler — Optional pluggable module for ephemeral agent provisioning. Four backends are configurable: containers (Docker/Podman), bare-metal processes, Firecracker microVMs, and the event backend, which performs no local compute — it emits reserved scale-up / scale-down events that a customer-authored provisioning workflow consumes to boot and tear down a cloud instance. Spawns agents on demand when no matching agent is connected, with label-based routing, two-level capacity limits (global + per-backend), warm pools, YAML configuration (scalers.d/ directory support), and live reload (kici-admin scaler reload across the cluster, or SIGHUP). Disabled by default — orchestrator works without it.
Independent database — Has its own PostgreSQL database separate from the Platform. Stores execution runs/jobs/steps, dispatch queue, webhook secrets, dedup cache, and scaler state. The orchestrator’s execution_runs and execution_jobs are the authoritative source of truth. The Platform receives execution status updates via WebSocket messages (execution.status, job.status.forward).

Source: packages/orchestrator/src/pipeline/processor.ts (webhook pipeline), packages/orchestrator/src/cluster/ (P2P coordination), packages/orchestrator/src/scaler/ (auto-scaler module), packages/orchestrator/src/server.ts (Platform/hybrid entry point)

Agent (`@kici-dev/agent`)

The agent is the execution worker. It runs on customer infrastructure and has full access to customer code.

Repository cloning — Clones the target repo with token-based auth (token in HTTP headers, not URLs, to prevent leakage).
Git credential helper — Registers a credential helper for the job’s git operations. Every network operation asks the orchestrator’s broker for a credential, so a token is minted seconds before use rather than held for the life of the job. Write access is opt-in and time-boxed: ctx.repo.withWrite(...) adds a repository-scoped grant for the duration of its callback and revokes it afterwards, with a TTL backstop.
Step execution — Runs steps in declaration order with full StepContext (zx shell, logger, environment, workflow/job metadata). Steps wrapped in a parallel() group run concurrently behind a maxParallel window, and each child reports as its own observable step with its own logs, status, timing, and retry.
Execution sandboxes — Runs the workflow runner as a separate child process with a sanitized environment, in one of three sandboxes: bare metal (process fork, with optional bubblewrap namespace isolation), a container runtime (the whole job lifecycle runs inside a disposable container), or inside a Firecracker microVM. Agent-internal credentials never reach customer workflow code.
Log streaming — Chunked log streaming back to the orchestrator with configurable size limits.
Dependency caching — Packs, uploads, and restores installed workflow dependencies so repeat runs skip the install step.
Graceful shutdown — SIGTERM with 10s grace period, SIGUSR1 for drain mode.

Source: packages/agent/src/execution/job-runner.ts (job lifecycle), packages/agent/src/execution/sandbox/ (execution sandboxes and the parallel step scheduler), packages/agent/src/server.ts (entry point)

Supporting packages

`@kici-dev/engine`

Shared business logic used by all three tiers. Single source of truth for cross-tier concerns. Has no internal @kici-dev/* dependencies — only a handful of third-party libraries.

Protocol message schemas (Zod-based, direction-specific unions including dashboard REST-over-WS, browser live streaming, the test-relay control plane, log pull, run events, peer-to-peer, cluster join, and source registration)
Provider interfaces (WebhookNormalizer, LockFileFetcher, ChangedFilesFetcher, FileContentsFetcher, CloneTokenProvider, RepoUrlBuilder, CheckStatusPoster)
Git credential vocabulary (forge names plus the credential reference, grant, request, and result shapes the SDK declares and the orchestrator’s broker resolves) and the agent→orchestrator relay protocol its credential helper calls. See Git credentials
Trigger matching engine (branch, path, event evaluation) and the changed-files diff range (the git range an event changed, shared by the orchestrator’s changed-files fetchers and the agent’s clone diff)
Content-requirement matcher (the declarative requires filter — pure data describing a query over the bytes of one source file at the event’s ref, interpreted by the orchestrator via the FileContentsFetcher so no author code runs there) and the shared text-match vocabulary (contains / notContains / matches / notMatches) it shares with the commit-message trigger filter
Dispatch inputs (input descriptors, extraction from the trigger event, and coercion to typed values)
Matrix expansion and fanout (combination expansion with include/exclude, job-name suffix formatting, and materialization of one matrix or multi-host job into N dispatchable children)
Execution status vocabulary (run/job/step status enums + terminal-state sets; lifecycle owned by the orchestrator’s execution tracker) and its presentation layer (the total precedence order that decides which status wins a roll-up, canonical-status resolution, and the per-status failure classification every consumer asks about)
Job-kind discriminator, alongside the status enums. It separates a standard job running steps from an invoke gate and from the proxy job that mirrors a summoned run
Check mode (the idempotent run modes apply / check / check-fail-on-drift and the per-step outcome vocabulary)
Webhook signature verification (HMAC-SHA256, timing-safe)
WebSocket close codes (unified across all tiers)
WebSocket rate limiting (WsRateLimiter)
Environment allowlist (safe env var filtering)
Secrets management (secret context resolution)
Context model (scoped secrets, ordered context merge, protection gates)
Approval requirements (normalized approver clauses shared by the orchestrator gate, the resolver, the held-run store, and the agent step round-trip)
Build provenance (in-toto statement schema, DSSE envelope, attestation bundle, verification)
Artifact name contract (the shared filesystem/URL-safe name schema the orchestrator, agent, and SDK all validate against)
Developer MCP tool schemas (argument schemas for the AI-agent tool surface) and the untrusted-content fence that wraps every repository- or contributor-supplied value an agent reads in a per-response random nonce, so log lines and error text cannot be read as instructions
Developer-operations contract (one row per workflow-developer operation declaring which entrypoints expose it — the shared REST API behind the web UI and the kici CLI, the AI-agent tool surface, and a curated UI flag — asserted against each real surface by congruence tests)
Repository-pattern negation check (the one verdict on whether a repository glob reads as a negation, so an allow or deny list entry never matches its own complement)
Permission matrix (the resource and permission-level vocabulary and the hasPermission check, shared by the Platform, which enforces it, and the browser dashboard, which shows or hides each control from it)
Run id references (parsing a full run id or a prefix of four or more hex characters, shared by the kici CLI, kici-admin and the Platform)
Label utilities (platform label derivation, runsOn normalization, kici:* set-only reserved namespace, role labels)
Host inventory (the canonical queryable host-roster schema shared by the orchestrator’s roster store, the agent-facing inventory API, and the SDK’s ctx.kici.inventory)
Audit policy and retention (per-action access-log sampling, warm-retention windows for cold-store eligibility, federated activity row schema)
Scaler backend type enum (container, bare-metal, firecracker, kubernetes, event; the orchestrator config rejects kubernetes) and the reserved event-name prefixes: kici. for KiCI system events and __ for events the orchestrator mints for itself, so a user step cannot forge either
Job resource vocabulary (the requests/limits shape the SDK accepts, the compiler validates and emits, the orchestrator uses for capacity math and kernel-side enforcement, and the dashboard displays)
Registration trigger type enum (registerable trigger discriminator)
Sandbox capability set (the Linux capability names a container sandbox may add or drop, shared by the SDK validator, the compiler, and the dispatch resolver)
Plan tier and subscription-status vocabulary (the hosted plan tiers, the purchasable subset, and the billing subscription states, shared by the Platform and the browser dashboard)
Infrastructure alert vocabulary (the diagnostics alert types and severities the Platform mints and the dashboard and kici CLI render)
Metric catalog (the generated Prometheus metric inventory, its naming policy, and metric-kind compatibility checks)
Bundler config (the shared workflow-bundle configuration factory on the barrel; the agent runtime uses the @kici-dev/core/ts-loader-hook to transform TypeScript on import, so no runtime path bundles a workflow)

Source: packages/engine/src/

`@kici-dev/sdk`

User-facing SDK for defining workflows in TypeScript. Provides factory functions (workflow(), job(), step()), trigger builders (pr(), push()), rules (rule(), skip()), matrix utilities, and DAG validation.

Source: packages/sdk/src/

`@kici-dev/compiler`

CLI tooling for workflow authors. Compiles .kici/workflows/*.ts to .kici/kici.lock.json, provides watch mode, local test execution, project initialization, and pre-commit hook integration.

It also holds the kici CLI’s remote commands: login, organization and orchestrator selection, run inspection, reruns and cancellation, approvals, notifications, and diagnostics. These commands call the hosted Platform’s dashboard API, the same API the web UI uses.

It also runs the local dev plane — an on-demand, fully local execution stack (embedded PostgreSQL, an orchestrator process, and a bare-metal-scaled agent) that lets an author run a workflow end-to-end on their own machine. That is why the compiler depends on @kici-dev/orchestrator and @kici-dev/agent: it resolves and spawns their built entry points rather than reimplementing them. See Local dev plane.

Source: packages/compiler/src/ (local-plane/ for the local dev plane)

`@kici-dev/core`

Light shared utilities with no server-side dependencies. It provides JSON-structured logging, error helpers, async-local-storage request context, and human-readable formatting (formatBytes/formatDuration/formatUptime). It also provides cryptographic helpers (sha256/sha256File/deriveSharedSecret plus symmetric encrypt/decrypt), retry-backoff computation, the shared diagnostics-result contract, and the redaction primitives for diagnostic bundles (redactConfig for structured config, scrubText for free text). The rest of its surface ships as subpath entry points: the temp-directory allocator and its garbage collector, package-manager detection, and CI-environment detection. Another subpath holds the idempotent-step runner, the check / confirm / apply primitive behind idempotent steps. A tar-packing subpath makes each hard-linked file pack as its own plain file entry, so packing a pnpm-installed node_modules tree cannot stall. The barrel also carries POSIX shell quoting (shQuote), OpenSSH file-list option quoting (sshFileListOption), and release-version ordering (compareReleaseVersions). It defines the compile schema version and the one compile-schema mismatch message every surface prints (formatCompileSchemaMismatch). It carries ConfigValidationError, which names the environment variable behind each failed configuration field, and makeCliStdioBlocking, which makes a CLI finish writing its output before it exits into a pipe. One further subpath holds the .kici/ source digest: the single content-hash definition the compiler writes into the lock file and the agent recomputes as its drift gate. It defines the anchor prefixes (REPO_ANCHOR/HOME_ANCHOR) inside cache and artifact archives, shared by the agent’s cache engine and the CLI’s artifact extractor. It also owns the published docs host: docsUrl() builds every docs link a CLI prints or a template scaffolds from one DOCS_SITE_URL constant, so no other package spells the host. Finally it supplies zx initialization (initZx()) and the TypeScript loader hook that transforms TypeScript on import. It is the dependency-light core that the SDK, compiler, and kici CLI consume directly so they stay free of heavier server-only dependencies. @kici-dev/shared re-exports it, so existing @kici-dev/shared import paths keep working.

Source: packages/core/src/

`@kici-dev/shared`

Shared utilities used across packages, including everything from @kici-dev/core (re-exported) plus server-side helpers. From core it carries initZx() for zx initialization, createLogger() for JSON-structured logging with TTY-aware formatting, and requestContext/getRequestContext()/enrichRequestContext() for async local storage request context. It also carries formatBytes/formatDuration/formatUptime for human-readable formatting and sha256/sha256File/deriveSharedSecret for cryptographic utilities. Its own server-side helpers are createPool()/createDb() for typed PostgreSQL connections and createMetricsRoutes()/createHealthRoutes() for HTTP route factories (Prometheus metrics and health endpoints). It adds RingBuffer for bounded collections, getReconnectDelay() for exponential backoff, and initTelemetry/createMeter for OpenTelemetry integration. setupGracefulShutdown runs a coordinated service shutdown with ordered steps. Its shutdown helpers stopHttpServer and closeDatabase stop the HTTP server and drain the connection pool, each within a time limit. It provides secret encryption (encrypt/decrypt/deriveKey) and the direct-database helpers the admin CLIs share (ensureDatabase and the *Direct queries). Its schema fingerprint (fingerprintSchema/diffFingerprints) reads a live PostgreSQL schema and compares it with the schema the release expects, which is what kici-admin db schema-diff reports. It also provides an S3 client factory (createS3Client) and the cold-store framework (BaseColdStore), which archives aged table rows to object storage in chunks and reads them back.

Source: packages/shared/src/

Dashboard

Web UI for KiCI. A browser single-page application that provides the operator dashboard with execution run listing, run detail views, real-time log streaming, settings management, and keyboard shortcut support. Authenticates via OIDC against the identity provider and communicates with the Platform REST-over-WebSocket API.

`kici` (wrapper)

Unscoped wrapper package that provides the kici CLI command. Its kici binary imports runCli() from @kici-dev/compiler/cli and runs it, so users can install kici globally or use it via npx kici.

Source: packages/kici/

`kici-admin` (admin CLI wrapper)

Unscoped wrapper package that ships two binaries: kici-admin, which runs runCli() from @kici-dev/orchestrator/cli for orchestrator administration tasks, and kici-agent, which imports @kici-dev/agent/server to start an agent. It therefore depends on both @kici-dev/orchestrator and @kici-dev/agent (the KICIADMIN → AGENT edge in the graph below).

Source: packages/kici-admin/

Package dependency graph

The following diagram shows how @kici packages depend on each other. Solid arrows are direct dependencies; dashed arrows are peer or dev dependencies (labeled).

Leaf packages (no @kici dependencies): @kici-dev/core and @kici-dev/engine. These can be tested and built independently. @kici-dev/shared builds on @kici-dev/core (which it re-exports) and on @kici-dev/engine for shared vocabularies. The dashboard depends on @kici-dev/engine for shared types (protocol schemas, execution status enums) and imports the Platform’s API type definitions as a dev dependency, but communicates with backend services at runtime via HTTP/WebSocket, not at compile time.

Runtime tiers (Platform, orchestrator, agent) all depend on @kici-dev/engine for shared business logic and @kici-dev/shared for utilities. Only the agent depends on @kici-dev/sdk (it loads workflow definitions at runtime).

The COMPILER → ORCH and COMPILER → AGENT edges exist solely for the local dev plane: the compiler spawns a local orchestrator and agent so an author can execute a workflow end-to-end without any deployed infrastructure. Nothing in the compile path itself reaches into either tier.

Connection overview

KiCI uses three WebSocket layers for real-time communication.

Platform ↔ Orchestrator

The orchestrator connects outbound to the Platform WebSocket endpoint. After authentication (API key validated via SHA-256 hash lookup), the connection is used for webhook relay, execution telemetry (events, status, logs), source registration, and peer discovery. Peer discovery is matchmaking only: the Platform pushes a peer.update membership list to every orchestrator sharing a routing key, and the orchestrators then connect to each other directly. Inter-orchestrator traffic such as job.reroute never transits the Platform.

Orchestrator ↔ Orchestrator (P2P)

When multiple orchestrators are deployed, they establish direct WebSocket connections to each other on the /ws/peer endpoint. Peers are discovered via the Platform matchmaker (Platform/hybrid modes) or static configuration (KICI_CLUSTER_PEERS env var, independent mode). Each connection first sets up an ECDH-encrypted channel. A worker, or a coordinator given a token, then authenticates with a join token, and the coordinator it connects to issues it a persistent peer credential. A join token binds to the first instance that uses it, and another instance is refused. A coordinator without a token issues its own credential from the shared database on its first peer connection. Later connections prove possession of that credential. Authentication is mutual: each side proves the shared credential or join token with a proof bound to the handshake. No credential or token crosses the wire, and the dialling side accepts nothing until the side it dialled has proved itself. A peer address the Platform announces is a hint until that proof succeeds. Traffic includes agent inventory heartbeats, job rerouting, progress reporting, cancel propagation, and Raft leader election. These messages never transit the Platform tier.

See Multi-orchestrator architecture for clustering details and Protocol messages for message schemas.

Orchestrator ↔ Agent

The agent connects outbound to the orchestrator WebSocket endpoint. After registration (agent ID, labels, concurrency), the connection is used for job dispatch, status reporting, and log streaming.

See Protocol messages and Webhook delivery for detailed message flows and schemas.

Authentication and multi-tenancy

KiCI uses application-level tenant isolation. The Platform dashboard API accepts three authentication methods (PATs, API keys, JWTs) and enforces org membership on every /api/v1/orgs/:customerId/* request.

Architecture overview

Three-tier relay model

Component responsibilities

Platform

Orchestrator (@kici-dev/orchestrator)

Agent (@kici-dev/agent)

Supporting packages

@kici-dev/engine

@kici-dev/sdk

@kici-dev/compiler

@kici-dev/core

@kici-dev/shared