# KiCI Architecture overview This bundle covers: How the runtime works: three-tier relay model, data flows, configuration. ## Configuration architecture Source: https://docs.kici.dev/architecture/configuration/ This document describes the internal design of the orchestrator's configuration management system. For operator-facing documentation, see [Configuration Reference](https://docs.kici.dev/operator/orchestrator/configuration/) and [Config Management Guide](https://docs.kici.dev/operator/orchestrator/config-management/). ## Config type system The configuration is modeled as three distinct types that merge into a final application config: ### LocalConfig Per-orchestrator settings loaded from a YAML file. These are instance-specific and never shared: ```typescript interface LocalConfig { database: { url: string }; instance?: { id?: string; mode?: 'platform' | 'hybrid' | 'independent' | 'observed' }; server?: { port?: number; basePath?: string; logLevel?: string }; scaler?: { configPath?: string; configDir?: string }; } ``` **Key property:** Every field except `database.url` is optional. An orchestrator can run entirely from env vars with no YAML file. ### SharedConfig Settings stored in the PostgreSQL `config_versions` table, written and read by the `/admin/config` routes and their `kici-admin config` commands: ```typescript interface SharedConfig { platform?: { url?: string; token?: string }; storage?: { type?: 's3'; bucket?: string; ... }; agentAuth?: 'token' | 'none'; agentTokenTtlMs?: number; queue?: { maxDepth?: number; timeoutMs?: number }; lockfileCache?: { max?: number; ttlMs?: number }; staleDetector?: { scanIntervalMs?: number; ... }; secrets?: { key?: string; keyFile?: string; bootstrapAdminToken?: string }; pgCustomerSecrets?: boolean; cluster?: { joinToken?: string; raftElectionTimeoutMinMs?: number; ... }; // ... tuning fields } ``` **Key property:** All top-level fields are optional. The DB may store a partial config. ### AppConfig The merged result type used throughout the codebase. Combines `LocalConfig` + `SharedConfig` with resolved defaults: ```typescript interface AppConfig { instanceId: string; // From local config or auto-generated mode: 'platform' | 'hybrid' | 'independent' | 'observed'; databaseUrl: string; // Flattened from database.url port: number; // Flattened from server.port basePath: string; platformUrl?: string; // Flattened from platform.url platformToken?: string; agentAuth: 'token' | 'none'; // With defaults applied queueMaxDepth: number; // Flattened from queue.maxDepth cluster: { instanceId: string; credentialFile: string; autoRotateCredentials: boolean; peers: string[]; ... }; // ... all other fields with defaults } ``` **Key property:** `AppConfig` uses flat field names (e.g., `databaseUrl` instead of `database.url`) for backward compatibility with the existing codebase. A `flattenToAppConfig()` function handles the mapping. ### How they merge `resolveFullConfig()` takes a `LocalConfig` and an optional `SharedConfig` and merges them: ``` defaults (getDefaults()) | v SharedConfig (argument) ──deepMerge──> merged layer 1+2 | v LocalConfig (from YAML) ──deepMerge──> merged layer 1+2+3 | v Env var overrides ──apply──> merged layer 1+2+3+4 | v flattenToAppConfig() ──flatten──> flat AppConfig shape | v appConfigSchema.safeParse() ──validate──> typed AppConfig ``` The `deepMerge` function merges objects recursively, replaces arrays (does not merge item-by-item), and skips `undefined`/`null` source values (they do not override existing values). **The `SharedConfig` argument is `null` in the shipped wiring.** `ConfigReloader` is the only non-test caller of `resolveFullConfig()`, and it is constructed with `sharedStore: null`. So the DB layer is skipped and the effective chain is defaults → YAML → env. The `config_versions` table is read by the `/admin/config` write and inspection routes, by `kici-admin rotate-key`, and by the cluster join flow — never by a running orchestrator's own config. ## Resolution chain ### Startup `server.ts` and `standalone.ts` both call `loadConfig()`, which parses `KICI_*` environment variables against the flat schema in `config.ts`. No YAML file and no database row participates: ``` Process start | v loadConfig() -> envDef.parse(process.env) -> AppConfig | v Connect to PostgreSQL, run migrations | v Start server (HTTP, WS, scaler, cluster) ``` The database URL therefore has to be an environment variable: the orchestrator needs it to reach PostgreSQL, and the shared config lives in PostgreSQL. ### Reload `resolveLocalConfig()` and `resolveFullConfig()` run on the reload path, not at startup: ``` SIGHUP / POST /admin/config/reload / kici-admin config reload | v resolveLocalConfig() -> YAML file + KICI_ env overlay | v resolveFullConfig(local, null) -> defaults -> YAML -> env -> AppConfig | v Hold databaseUrl, port, instanceId and storage at their startup values | v Atomic swap into ConfigReloader.currentConfig ``` ### Env var processing Environment variables are processed in two stages: 1. **Direct mappings:** `KICI_DATABASE_URL` -> `database.url`, etc. A lookup table in `env-overlay.ts` maps known env var suffixes to config path arrays. 2. **Multi-app GitHub provider:** `KICI_PROVIDERS_GITHUB__` is parsed by stripping the `PROVIDERS_GITHUB_` prefix, finding the field suffix (`APP_ID`, `PRIVATE_KEY`, `WEBHOOK_SECRET`), and deriving the app name from the middle segment. App names are lowercased with underscores converted to hyphens (`MAIN_ORG` -> `main-org`). Type coercion is applied based on known field types: numeric fields are parsed as numbers, boolean fields are compared against `"true"`, all others remain strings. ## DB schema ### config_versions table ```sql CREATE TABLE config_versions ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), version SERIAL NOT NULL UNIQUE, config JSONB NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), created_by TEXT NOT NULL, description TEXT, encrypted_paths TEXT[] NOT NULL DEFAULT '{}' ); CREATE INDEX idx_config_versions_version ON config_versions(version DESC); CREATE INDEX idx_config_versions_created_at ON config_versions(created_at DESC); ``` **Design choices:** - **SERIAL version:** Auto-incrementing integer provides a total ordering of config changes. Simple to compare in heartbeats. - **JSONB config:** Stores the full shared config document. JSONB allows future querying/indexing if needed, though we always read the full document. - **Immutable rows:** Each change creates a new version. Old versions are never modified, providing a full audit trail. - **encrypted_paths:** Array of concrete dot-separated paths (e.g., `platform.token`, `cluster.joinToken`) that contain encrypted values. Stored alongside the config so the system knows exactly which fields to decrypt without runtime path list dependency. - **created_by:** Identifies the source of the change (e.g., `cli:seed`, `api:set`, `api:rollback`). ### Versioning strategy - Version numbers are auto-incrementing integers managed by PostgreSQL SERIAL - Rollback creates a new version (copy of target) rather than reverting to the old version number - Example: versions 1, 2, 3 exist. Rollback to 1 creates version 4 with the content of version 1 - This preserves the audit trail: version 4 records when and why the rollback happened ## Encryption ### Algorithm AES-256-GCM via the existing `secrets/crypto.ts` module: - **Key:** 32-byte AES-256 key derived from the master key (`KICI_SECRET_KEY`) - **IV:** Random 12-byte initialization vector per encryption - **Auth tag:** 16-byte GCM authentication tag - **AAD:** `config-field:` (e.g., `config-field:platform.token`) -- binds ciphertext to its specific location - **Wire format:** base64(IV || AuthTag || Ciphertext) - **Key version:** Integer stamp for the master-key generation that sealed the row. Every new row is written under the active generation (hydrated from `MAX(key_version)` at startup). `kici-admin rotate-key` bumps the stamp atomically — the decrypt path accepts the current generation, and during the grace window also the previous one (`KICI_SECRET_KEY_OLD`), so historical rows and rollbacks continue to work seamlessly across rotations. ### Sensitive field paths The following glob patterns define sensitive fields: ```typescript const SENSITIVE_FIELD_PATHS = [ 'platform.token', 'secrets.key', 'secrets.bootstrapAdminToken', 'cluster.joinToken', ] as const; ``` ### Encryption flow ``` Save: config -> resolveGlobPaths(SENSITIVE_FIELD_PATHS) -> for each concrete path: encrypt(value, key, "config-field:") -> store { encrypted_config, encrypted_paths[] } Load: row -> for each path in encrypted_paths: decrypt(value, key, "config-field:") -> SharedConfig Export (redacted): row -> decrypt (if master key available) -> replace encrypted_paths values with "***REDACTED***" ``` ### Rollback optimization When rolling back, the target version's encrypted config is copied as-is to the new version. No re-encryption is needed because: - The same master key applies (all orchestrators share the same key) - The same AAD applies (paths are identical) - The `encrypted_paths` array is preserved from the target version ## Hot-Reload ### ConfigReloader design The `ConfigReloader` class manages the full reload lifecycle: ```mermaid flowchart TD SIGHUP["SIGHUP (signal)"] --> trigger HTTP["HTTP POST /reload"] --> trigger Cluster["Cluster auto-fix"] --> trigger["triggerReload()
500ms debounce"] trigger --> execute["executeReload()
boolean mutex"] execute --> resolveLocal["resolveLocalConfig()"] execute --> getLatest["getLatest (DB)
skipped: sharedStore is null"] execute --> resolveFull["resolveFullConfig()
(merge + validate)"] resolveLocal --> check["Check restart-required fields"] getLatest --> check resolveFull --> check check --> swap["Atomic swap
onConfigApplied()"] swap --> onProvider["onProviderChange (if changed)"] swap --> onScaler["onScalerReload (always)"] swap --> onPlatform["onPlatformReconnect (if changed)"] ``` ### Safety guarantees - **Mutex:** Boolean flag prevents concurrent reloads. Second reload returns `{ success: false, errors: ["Reload already in progress"] }`. - **Debounce:** Rapid triggers (e.g., multiple SIGHUP signals) are collapsed into a single reload with a 500ms window. - **Validation before swap:** The new config must pass full schema validation. On failure, the old config is preserved and an error is logged. - **Restart-required detection:** `databaseUrl`, `port`, `instanceId` and `storage` are compared. If changed, the old values are preserved in the applied config and a warning is logged. - **No crash on failure:** The orchestrator always keeps running with the old config if anything goes wrong during reload. ### Subsystem callbacks The `ConfigReloader` uses a dependency injection pattern with callbacks for subsystem re-initialization: | Callback | When Called | Purpose | | --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `onProviderChange` | Provider config changed | Reserved callback. Providers are DB-managed via the sources table, so the change detector always reports no change | | `onScalerReload` | Always on successful reload | Reload scaler YAML config, from the path the process started with | | `onPlatformReconnect` | Platform URL or token changed | Logs that the Platform connection settings changed. The connection is not re-established; `standalone.ts` registers no handler | | `onConfigApplied` | Always on successful reload | Atomic config reference swap, increment local config version | ### Prometheus metrics | Metric | Type | Labels | Description | | ------------------------------- | ------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `kici_orch_config_reload_total` | Counter | `result` (attempted/success/failed), `source` (sighup/http/cluster/cli) | Config reload attempts and outcomes | | `kici_orch_config_version` | Gauge | -- | Shared config version from the DB. Set only when the reload path reads a version, so it carries no value today | ## Multi-Provider ### ProviderRegistry The `ProviderRegistry` maps routing keys to provider bundles. Each routing key (e.g., `github:12345`) is associated with a `ProviderBundle`. Only the normalizer is required; every other capability is optional, so a bundle carries exactly the interfaces its provider implements: - `WebhookNormalizer` (required) -- normalizes incoming webhooks to a standard format - `LockFileFetcher` -- fetches lock files from the repository - `ChangedFilesFetcher` -- determines which files changed - `FileContentsFetcher` -- reads arbitrary repository files at a ref, for the declarative content-requirements (`requires`) filter - `CloneTokenProvider` -- generates clone tokens for agents - `RepoUrlBuilder` -- builds clone URLs and raw file URLs - `CheckStatusPoster` -- posts check statuses (approval/hold) to the git provider A bundle also carries a `hasForkModel` flag, set for a provider whose head ref can live outside the base repository. It is what admits a pull-request event to the org fork switch. GitHub sets it; a generic source (whose trust boundary is its verification secret) and a local source (whose trust boundary is on-disk ownership) do not. A GitHub App source populates all seven. The file-contents capability arrives as a per-delivery factory rather than a prebuilt instance: a GitHub client is scoped to one installation, and the installation id is known only once the delivery's credentials are resolved. A plain generic webhook source carries only the normalizer, because it has no repository API to fetch a lock file or post a check against. The pipeline skips the stages whose interface is absent rather than failing the delivery. Provider registrations are managed via the `sources` database table, not via `SharedConfig`. When the orchestrator connects to the Platform relay, it reads source records from the DB and sends `source.register` messages. Changes to sources (add/remove) are detected via PostgreSQL LISTEN/NOTIFY on the `sources_change` channel and pushed to the Platform via `source.secrets` and `source.register`/`source.deregister`. ### Per-App Credentials Each source record contains its own `appId` and `privateKey` (stored as scoped secrets). When processing a webhook, the orchestrator looks up the routing key to find the matching source and uses its credentials for JWT generation, clone tokens, and check run updates. ## Cluster sync ### Heartbeat config version In clustered deployments, each orchestrator includes its config version in Raft heartbeat metadata via the `configVersion` optional field on the `peerHeartbeatSchema`. **That number is a local reload counter, not a shared config version.** `onConfigApplied` increments it on every successful reload and publishes the new value to the peer registry. It counts how many times this instance has reloaded. When the `PeerRegistry` processes a heartbeat: 1. Compare `localConfigVersion` with `peer.configVersion` 2. If `peer.configVersion > localConfigVersion` AND both are > 0: - Invoke the `onConfigVersionBehind` callback - This triggers a config reload, which re-reads the environment and the local YAML file ### Auto-remediation flow ``` Orchestrator A (reloaded 5x) Orchestrator B (reloaded 3x) │ │ │──── heartbeat(configVersion=5) ────>│ │ │ │ compare: 5 > 3 │ trigger reload │ │ │ resolveFullConfig(local, null) │ -> counter becomes 4 │ │ │<── heartbeat(configVersion=4) ──────│ │ │ ``` B converges on A's count only after it has reloaded as many times as A has. Each instance reads its own environment and its own YAML file, so the two agree on content only when those inputs agree. ### Guard conditions - Version comparison only triggers when **both** local and peer versions are > 0 - This prevents false triggers from: - Orchestrators that do not report `configVersion` (field is optional, defaults to 0) - Newly started orchestrators before their first reload ## See also - [Configuration Reference](https://docs.kici.dev/operator/orchestrator/configuration/) -- operator guide - [Config Management Guide](https://docs.kici.dev/operator/orchestrator/config-management/) -- CLI and API guide --- ## Data flows Source: https://docs.kici.dev/architecture/data-flows/ This document describes the key data flows through the KiCI architecture: webhook delivery, job execution, developer-initiated remote runs, dependency caching, re-run and cancel, trace ID propagation, internal event routing, and generic webhook ingestion. > **Lock file schema version:** The orchestrator accepts a compatibility window of lock schema versions rather than an exact match. A lock is accepted when its `schemaVersion` is at or above the orchestrator's oldest supported version (additive bumps add fields older readers ignore) and the orchestrator's own schema is at or above the lock's `minReaderVersion` (the newest breaking version at compile time). A lock below the floor must be recompiled with `kici compile` and pushed; a lock requiring a newer reader means the orchestrator must be upgraded. Both out-of-window cases are rejected with an actionable error rather than a silent mis-route. See [lock file and drift](https://docs.kici.dev/user/lock-file-and-drift/#schema-compatibility-window). ## Webhook delivery flow A webhook event from a provider (e.g., GitHub) travels through three tiers before execution begins. ``` GitHub --> Platform Relay --> Orchestrator --> Agent 1. Webhook 2. WebSocket 3. Job dispatch POST relay + execution ``` ### Step by step 1. **Provider sends webhook** to the Platform relay endpoint. 2. **Platform routes the webhook** to the right orchestrator over WebSocket and forwards the body bytes verbatim. Platform never sees customer HMAC secrets — signature verification happens entirely on the orchestrator after reassembly. 3. **Orchestrator admits the delivery**, then **verifies the signature** (HMAC-SHA256 against per-source webhook secret, with dual-secret rotation support). Admission runs first, on the routing key alone: when the ingest admission controller sheds, the orchestrator records an `event_log` breadcrumb with status `shed` and ACKs `shed_retry_later`, which the Platform answers as **429** with `Retry-After`. See [ingest admission shed](https://docs.kici.dev/architecture/webhooks/webhook-delivery/#ingest-admission-shed-step-3). 4. **Orchestrator dedup check** against dual-layer `DedupCache` (in-memory set + `dedup_cache` DB table). 5. **Orchestrator resolves provider** by looking up the provider bundle from the `ProviderRegistry` using `getByRoutingKey()` (exact match first, falls back to provider type prefix for backward compatibility). Skips processing if the provider is unknown. 6. **Orchestrator normalizes** the webhook via the provider's `WebhookNormalizer` (extracts branch, event type, action, sender). 7. **Orchestrator extracts repo and credentials** from payload (repository identifier from `repository.full_name`, provider credentials such as GitHub installation ID). 8. **Orchestrator handles /kici commands** in `issue_comment` events: intercepts `/kici approve` and `/kici reject` approval commands before trigger matching, delegating to `handleApprovalComment()` for security hold management. 9. **Orchestrator resolves trust** for PR events (determines lock file source: head vs base branch). 10. **Orchestrator fetches lock file** via the provider's `LockFileFetcher` (cached with LRU). For untrusted PR events, fetches both base and head lock files in parallel; for trusted PRs and pushes, fetches from head SHA. 11. **Orchestrator detects workflow modifications** for untrusted PR events by comparing base and head lock files via `detectWorkflowModifications()`, applying security holds when non-trusted contributors modify workflow files. 12. **Orchestrator extracts registrations** on default-branch pushes: persists registerable workflows (event, schedule, lifecycle triggers) for cluster-wide event matching. 13. **Orchestrator notifies the event router** on default-branch pushes: after the registrations are persisted, emits a `registration.updated` event via `eventRouter.emit()` (if event routing is active). Workflow event subscriptions are the persisted registrations themselves, matched at emit time through the registration index. 14. **Orchestrator fetches changed files** via the provider's `ChangedFilesFetcher` for path-based trigger filtering (skipped when no workflow uses path filters). 15. **Orchestrator matches triggers** against the lock file using `matchWorkflowsForEvent()` from `@kici-dev/engine` -- an event-type-bucketed candidate scan that evaluates only the workflows subscribed to this event type. (The single-registration global / cross-source paths evaluate one lock entry at a time via `matchAllWorkflows()`.) 16. **Orchestrator applies the content-requirements filter** to the matched candidates: for each trigger that declares `requires`, it reads the named source files at the event's ref through the provider's `FileContentsFetcher` (once per distinct `(repo, sha, path)` via an LRU cache) and evaluates the declarative requirement. Candidates that fail -- or that cannot be evaluated at all (unreadable or oversize content, a fetch error, no fetcher wired) -- are dropped before dispatch with the concrete reason logged. No workflow code runs at this stage. Skipped entirely when no matched trigger declares `requires`. 17. **Orchestrator checks caches** for source tarballs and dependency tarballs. 18. **Orchestrator dispatches jobs** to agents via the job queue and WebSocket. 19. **Orchestrator persists a delivery row** keyed by `(org_id, delivery_id)` to its own `event_log`, including a pointer to the gzipped payload in object storage. The orchestrator's delivery log is surfaced in the dashboard's Settings → Event log tab. See [`webhook-delivery.md`](https://docs.kici.dev/architecture/webhooks/webhook-delivery/#delivery-log). ## Job execution flow Once the orchestrator has matched triggers and resolved caches, jobs are dispatched to agents. ``` Orchestrator Agent Sandbox (child process) | | | |-- job.dispatch (WS) ------------>| | | (jobConfig, sourceTarUrl, | | | sourceTarHash, depsUrl, |-- Create sandbox ------->| | depsHash) | (container/bare-metal/ | | | firecracker) | | | |-- Restore .kici/ source (tarball) | | |-- Restore deps (tarball) | | |-- Load workflow (TS loader hook) | | |-- Evaluate rules | | |-- Execute steps | |<-- IPC (step status, ----| | | log lines, events) | |<-- job.status (WS) -------------| | | (step progress, completion) |-- Teardown sandbox ----->| | | | ``` ### Agent pipeline The agent delegates job execution to an `ExecutionSandbox` (container, bare-metal, or firecracker). The sandbox runs customer code in an isolated child process -- never in the agent's V8 isolate. Six job types are handled. Only the first uses a sandbox; the other five run in-process, because they never execute customer workflow steps: - **Execution jobs** -- the standard sandbox path. - **Init-only jobs** -- dynamic field resolution. - **Global-eval-round jobs** -- filters and generators for the candidate global workflows of one workflow repo. - **DynamicJobFn evaluation jobs** -- runtime job generation. - **Bring-up jobs** -- init-runner SSH bring-up. No clone, no sandbox. - **Build-only jobs** -- cache population. See [Job execution lifecycle](https://docs.kici.dev/architecture/execution/job-execution/) for details. 1. **Report running** -- Send `job.status: running` immediately upon accepting the dispatch 2. **Sandbox selection** -- Determine execution mode (container, bare-metal, firecracker) from job config and environment 3. **Sandbox setup** -- Create and start the execution environment (container: `docker create`/`start`; bare-metal: validate; firecracker: detect) 4. **Context emission** -- Send `job.context` to orchestrator with runtime details (Node version, OS, arch, sandbox type) 5. **Sandbox execution** -- The sandbox child process handles the inner pipeline: `.kici/` source tarball restore, deps tarball restore, workflow loading (dynamic-import `.ts` via the shared TypeScript ESM loader hook), step extraction, rule evaluation, step execution sequentially with timeout and abort support. There is no runtime bundling step — workflow TS is transformed on import, not ahead of time. 6. **IPC callbacks** -- Step status, log lines, event emissions, and concurrency reports flow from the sandbox to the agent via IPC, then to the orchestrator via WebSocket 7. **Report** -- Send final `job.status` back to orchestrator with step results and timing 8. **Cleanup** -- Tear down sandbox and remove work directory ## Remote run flow (`kici run remote`) A developer running `kici run remote` from a working tree initiates a run through the three-tier relay without a provider webhook. The flow splits into two independent planes: a **control plane** through the Platform relay, and a **data plane** that uploads the working-tree overlay straight to object storage. ``` Developer machine Platform relay Orchestrator Object store | | | | |-- upload-init (control) ----->|--- WS relay ----------->| | | (org, cluster, overlay | |-- mint presigned | | metadata, inline lock) | | PUT URL ---------->| |<-- presigned PUT URL ---------|<--- WS relay -----------| | | | | | |== overlay tarball PUT (data plane) ===========================================>| | | | | |-- trigger (control) --------->|--- WS relay ----------->|-- dispatch jobs | | | | (agents fetch | | | | overlay) | |-- poll logs + status -------->|--- WS relay ----------->| | |<-- log chunks + status -------|<--- WS relay -----------| | ``` ### Control plane Run initiation (`upload-init`), the trigger, status polling, log retrieval, and cancellation all flow from the developer machine to the Platform, which relays them over its WebSocket connection to the org's orchestrator. The developer machine never talks to the orchestrator's HTTP API directly. Logs are delivered by the CLI polling the Platform for log chunks — tracked by a monotonic line cursor — and run status until the run reaches a terminal state; there is no direct streaming socket between the developer machine and the orchestrator. ### Data plane The working-tree overlay tarball uploads **directly** from the developer machine to the orchestrator's object store via a pre-signed PUT URL minted during `upload-init`. The overlay bytes never pass through the Platform. Because of this split, only the object store needs to be reachable from the developer machine — the orchestrator can sit behind a private network. See [Storage layout](https://docs.kici.dev/operator/orchestrator/storage-layout/) for the upload-endpoint configuration. ### Org anchor The run is dispatched to the developer's active organization (selected with `kici org use`, or overridden per-run). The orchestrator anchors its bound organization with a system-managed **remote source** (routing key `remote:`) that it auto-provisions — no manual webhook source is required, so a zero-source org is immediately routable for remote runs. The Platform forces the run's routing key to `remote:` server-side; the developer never sets a routing key. When an org has more than one connected orchestrator cluster, the CLI selects the target cluster explicitly (or relies on the per-org default), and a single connected cluster is auto-selected. Remote runs are offered by the Platform; an orchestrator with no Platform connection cannot serve them. Executing workflow steps on the developer machine with no orchestrator is the separate `kici run --local` path. ## Source and dependency caching flow KiCI runs two orchestrator-side caches — the **source tarball cache** (raw `.kici/` directory minus `node_modules/`) and the **dependency tarball cache** (packed `node_modules/`). Both use a build-then-execute pattern: the orchestrator checks the caches before dispatching execution jobs, and if the source cache is cold a build agent populates both in one pass. With the shared TypeScript loader hook plus source tarball, execution agents never install or compile `.kici/` at runtime — they restore the workflow directory and its `node_modules` with two S3 GETs and extract. The workspace checkout is unaffected: an execution job still clones the repository at the dispatch ref unless it sets `checkout: false`, and the source tarball is restored over that clone. ### Cache miss flow When the source cache is cold and the dep cache is also missing: ``` Webhook | v Trigger Match | v Cache Check (source: MISS, deps: MISS) | v Build Job Dispatch --> Build Agent (kici:role:builder + matching kici:os:/kici:arch:) | | | |-- git clone + checkout SHA | |-- npm ci in .kici/ | |-- Pack .kici/ source (portable tar.gz, excludes node_modules) | |-- Pack .kici/node_modules (portable tar.gz) | |-- Upload source tarball to cache (source/v2/{orgId}/{sourceTarDigest}.tar.gz) | |-- Upload deps tarball to cache (deps/{plat}-{arch}/{depsHash}.tar.gz) | |-- Upload deps companion .hash file | |-- Report success (cache.upload.complete × 2) | | v v Build Complete <---------+ | v Get sourceTarUrl + depsUrl from cache (pre-signed S3 GETs) | v Execution Job Dispatch --> Execution Agent | | | |-- Download source tarball (sourceTarUrl) -> extract to workDir/.kici/ | |-- Download deps tarball (depsUrl) -> verify SHA-256 -> extract to .kici/node_modules/ | |-- Register @kici-dev/core/ts-loader-hook | |-- Verify workflow contentHash against lock file (drift guard) | |-- Dynamic-import workflow .ts | |-- Execute steps | |-- Report result | | v v Done <-----------------------+ ``` The execution agent never rebuilds `.kici/`. The source tarball IS the workflow repo's `.kici/` directory, extracted over the checkout. ### Cache hit flow When both caches have valid entries (the common case after the first run at a commit SHA): ``` Webhook | v Trigger Match | v Cache Check (source: HIT, deps: HIT) | v Get sourceTarUrl + depsUrl from cache (pre-signed S3 GETs) | v Execution Job Dispatch --> Execution Agent | | | |-- Download source tarball (sourceTarUrl) -> extract | |-- Download deps tarball (depsUrl) -> verify SHA-256 -> extract | |-- Register TS loader hook | |-- Dynamic-import workflow .ts | |-- Execute steps | |-- Report result | | v v Done <-----------------------+ ``` No build job is dispatched. The execution agent performs exactly two S3 GETs and extracts. ### Partial cache hit The source cache and dep cache are independent. Four combinations are possible: | Source | Deps | Behavior | | ------ | ---- | ------------------------------------------------------------------------------------ | | HIT | HIT | Direct execution dispatch (fastest, two S3 GETs) | | HIT | MISS | No build job; execution agent falls back to inline `npm ci` after restoring source | | MISS | HIT | Build job for source only (agent still packs deps opportunistically); then execution | | MISS | MISS | Build job for source + deps (single job packs both), then execution | Dep cache misses alone do **not** trigger a build job. Deps are platform-specific (`deps/{platform}-{arch}/{hash}.tar.gz`) so a build job would need a builder agent matching the target platform, which may not exist (e.g., an arm64 builder when only x64 builders are available). When the source cache misses, the dispatched build job piggy-backs dep packing if deps are also missing. A single build job handles both artifacts when both miss, avoiding duplicate builds. ### Cross-source / no-contentHash workflows - **Lock files without `contentHash`** (schema v1) skip the source cache entirely; agents compile from source. Regenerate lock files with `kici compile` to enable caching. The current lock file schema version is 41. - **Cross-source / global-workflow dispatch** (a workflow registered against source A fired by a webhook on source B) bypasses both caches. The registration's lock file entry still carries `contentHash`, but the cross-source path always clone-and-installs — the eval temp dir doesn't ship `@kici-dev/sdk`. The execution agent still verifies `contentHash` against the cloned source for drift detection. ### Build deduplication When multiple webhooks trigger simultaneously for the same repository state, the `BuildCoordinator` coalesces concurrent build requests using a combined key (`contentHash:lockfileHash`). Only one build job runs; all waiting dispatches share the result. ### Graceful degradation If cache storage is unavailable or a download fails: - **Source tarball download failure:** Hard failure today — the agent does not fall back to rebuilding `.kici/` from the checkout. In practice this is rare because the same orchestrator that issued the pre-signed URL controls the cache backend. - **Dep tarball download failure:** Agent falls back to running `npm ci` / `npm install` inline. - **Dep tarball hash mismatch:** Agent retries the download twice (3 total attempts), then fails the job (no fallback for integrity failures). - **Source tarball drift (extracted `contentHash` ≠ lock file):** Hard failure with "Lock file is out of date: workflow source changed without regenerating kici.lock.json" — see [Lock file and drift](https://docs.kici.dev/user/lock-file-and-drift/). - **Build failure:** Execution is skipped entirely with a "Build failed" check status. Workflows that contain dynamic job entries (DynamicJobFn) are allowed to proceed with their dynamic eval jobs since those compile from source. - **No cache configured:** Agent runs inline install for every job (pre-caching behavior). ## Cache storage architecture Both source and dep caches use `S3CacheStorage` as the sole backend. The `CacheStorage` interface provides a consistent API, but S3 (or any S3-compatible service: SeaweedFS, MinIO, LocalStack) is the only supported implementation. ``` +------------------+ | CacheStorage | | (interface) | +--------+---------+ | +---------+---------+ | S3CacheStorage | | (AWS S3, SeaweedFS,| | MinIO, LocalStack)| +--------------------+ ``` ### Cache key design Cache keys reflect that source tarballs and deps have different platform characteristics: - **Source:** `source/v2/{orgId}/{sourceTarDigest}.tar.gz`, with a `source/v2/{orgId}/{contentHash}.hash` pointer — platform-agnostic, and scoped to the owning organization so two repositories with matching `.kici/` trees never share one object. Raw TypeScript source is identical regardless of CPU architecture, so one entry is shared across all platforms. `contentHash` is the per-workflow hash from the lock file (`SHA-256(COMPILE_SCHEMA_VERSION + ":" + rawSource [+ "\0" + assetDigest])`, where `COMPILE_SCHEMA_VERSION = 7` and line endings are normalized to LF so the hash agrees across platforms). - **Deps:** `deps/{platform}-{arch}/{depsHash}.tar.gz`, with a `deps/{platform}-{arch}/{lockfileHash}.hash` pointer holding that hash — the tarball is addressed by its own content, so two builds sharing a lock file cannot leave a tarball and an integrity hash that disagree. Platform-specific. Native dependencies in `node_modules` differ across architectures, so each platform/arch combination gets its own cache entry. The orchestrator derives the target platform/arch for dep cache lookups by probing `AgentRegistry.findAvailable()` with the workflow's first job's `runsOn` labels to find a representative matching agent, then using that agent's platform and arch. Falls back to `linux/x64` if no matching agents are registered. ### TTL and eviction (touch-on-read) Both caches refresh TTL on read via `touch-on-read`. An entry's lifetime is reset every time an orchestrator issues a pre-signed GET URL for it. Default TTL is `KICI_CACHE_TTL_DAYS=30`; entries unused for 30 days expire at the storage level. Actively used sources and deps stay in cache indefinitely as long as they continue to be referenced by inbound webhooks or reruns. See [`docs/operator/dependency-caching.md`](https://docs.kici.dev/operator/dependency-caching/#cache-behavior) for configuration. For the full per-package bucket and prefix inventory — cache, logs, cold-store, and the observability sidecar buckets — see [orchestrator storage layout](https://docs.kici.dev/operator/orchestrator/storage-layout/). ### Pre-signed URL upload flow Agents upload artifacts directly to S3 using pre-signed PUT URLs. This eliminates the orchestrator as a data proxy — only coordination messages flow through WebSocket. ``` Agent Orchestrator S3 | | | |-- cache.upload.request ------->| | | { type: "source"|"dep", | | | key: "source/..." or | | | "deps/..." } | | | |-- getUploadUrl(key) ----->| | | (PutObject pre-sign) | |<-- cache.upload.response ------| | | { url: "https://s3.../..." } | | | | | |-- HTTP PUT (artifact body) --------------------------->| | (direct S3 upload) | | | | | |-- cache.upload.complete ------>| | | { type, key, depsHash? } |-- initMeta(key) -------->| | | (CopyObject to set | | | TTL metadata) | | |-- put(hashKey) --------->| | | (companion .hash file | | | for deps integrity) | ``` The two-phase metadata approach (`upload via PUT` then `initMeta via CopyObject`) works around the limitation that S3 pre-signed URLs cannot include custom metadata headers. For dependency tarballs, the agent also reports the SHA-256 content hash in `cache.upload.complete`; the orchestrator stores it as a companion `.hash` file alongside the tarball. When dispatching execution jobs, the orchestrator reads this hash and includes it as `depsHash` in `job.dispatch`, enabling agent-side integrity verification on download. Source tarballs do not use a companion `.hash` file — the workflow `contentHash` carried in `sourceTarHash` is used to verify the extracted source against the lock file after extraction, which covers drift end-to-end. ### URL delivery (downloads) Agents receive pre-signed S3 GET URLs (15-minute expiry) directly in `job.dispatch` messages. Agents download artifacts from S3, bypassing the orchestrator for all data transfer. ## User-facing cache flow The source/dep cache above is internal: the orchestrator owns its keys and decides when to hit or build. The **user-facing cache** is driven by the workflow author — the declarative `cache: { key, paths, restoreKeys? }` on a job/step, or the imperative `ctx.cache.restore()` / `ctx.cache.save()` API (see [SDK caching reference](https://docs.kici.dev/user/sdk/caching/)). It reuses the same object-storage backend and the same direct-to-storage presigned-URL transport, but the agent — not the orchestrator — initiates each restore and save over WebSocket. The agent's cache module archives `paths` into a gzipped tarball (computing a SHA-256 over the bytes) and streams downloads back through a checksum-verified extract pipeline. The orchestrator's `UserCache` owns the `cache////-` namespacing (the discriminator is a hash of the exact cache key, so two keys differing only by case stay two objects on a case-insensitive store), the immutable first-save check, the `restoreKeys` prefix scan, the two-phase atomic save, and per-org quota/TTL eviction. ### Restore flow ``` Agent Orchestrator (UserCache) Object storage | | | |-- cache.user.restore.request ------->| | | { key, restoreKeys? } |-- exact key in read prefixes ->| | | (isolated: iso// | | | then shared/; trusted: | | | shared/ only) | | |-- restoreKeys prefix scan ---->| | | (newest match wins) | | |-- getUrl(matched) + touch ---->| |<-- cache.user.restore.response ------| | | { hit, matchedKey?, | | | downloadUrl?, tarHash? } | | | | | |-- HTTP GET (tarball body) -------------------------------------->| | (direct download; verify tarHash, extract paths) | ``` The restore resolves the exact `key` across the ref's read prefixes first, then each `restoreKeys` prefix in order (newest matching entry wins). A trusted ref reads only `shared/`; an untrusted/fork ref reads its own `iso//` scope and then falls back to `shared/`. On a hit the response carries a presigned GET URL plus the tarball's `tarHash`, which the agent verifies before extracting. ### Save flow (two-phase atomic) ``` Agent Orchestrator (UserCache) Object storage | | | |-- cache.user.save.request --------->| | | { key } |-- has(final key)? ------------>| | | (immutable: skip if exists) | | |-- getUploadUrl(.tmp-) -->| |<-- cache.user.save.response ---------| | | { uploadUrl?, skip } | | | | | |-- HTTP PUT (tarball body) ----------------------------------->| | (direct upload to temp object) | | | | |-- cache.user.save.complete -------->| | | { key, tarHash, sizeBytes } |-- copy(temp -> final) -------->| | |-- delete(temp) --------------->| | |-- initMeta(final) ------------>| | |-- put(.hash) + put(.size) ---->| | |-- enforce per-org quota ------>| ``` The save is **immutable** and **atomic**. The orchestrator declines (`skip: true`) up front if the exact key already exists. Otherwise the agent uploads to a `.tmp-` object via a presigned PUT, then `cache.user.save.complete` triggers a server-side copy temp→final, a delete of the temp, an `initMeta` to stamp TTL metadata, and `.hash` / `.size` companion writes. Because the final key only appears after the copy, a crashed upload never leaves a corrupt committed entry. The committing save then enforces the per-org byte quota, evicting oldest entries until the org is back under `KICI_USER_CACHE_QUOTA_BYTES`. ### Trust → scope mapping The orchestrator threads a `cacheRefScope` onto each `job.dispatch`. A **trusted** ref (the repo's own branches, default branch) maps to the `shared` write scope; any other ref (a fork PR) maps to `isolated`, writing to a per-run `iso//` scope. This is the cache-isolation model: a fork can restore from the trusted `shared/` cache but can never write into it, so it cannot poison the entries a trusted branch later restores. The org segment of the key namespace (`cache//`) is the per-tenant boundary — no tenant can read another tenant's cache. See [orchestrator storage layout](https://docs.kici.dev/operator/orchestrator/storage-layout/#user-cache) for the full prefix map and quota/TTL knobs. ## Internal event routing flow Internal events (custom events from `ctx.emit()` and system events from workflow/job completion) flow through the event router for fan-out delivery to matching workflows. ``` Step ctx.emit('event-name', payload) | v Agent IPC (fork channel or stdout JSON-lines) | v Agent -> event.emit WS message -> Orchestrator | v EventRouter.emit() |-- CircuitBreaker check (chain depth, rate limit) — fail-fast, in-memory |-- BEGIN TRANSACTION | |-- EventStore.writeWith(tx) -> INSERT into kici_events table | |-- pg_notify('kici_event_channel', eventId) (queued; fires on commit) |-- COMMIT (rollback discards both insert and notify atomically) | v All Orchestrators LISTEN on 'kici_event_channel' channel | v EventRouter.onNotification(eventId) [private] |-- EventStore.tryLeaseForProcessing(eventId, nodeId, leaseDurationMs) | (atomic UPDATE: claim only if processed=false AND dlq_at IS NULL | AND (claimed_at IS NULL OR claimed_at < NOW() - leaseDurationMs); | increments attempts and records claimed_at/claimed_by atomically) |-- If lease acquired: | |-- processSubscriptions(event): | | |-- If RegistrationIndex available: | | | Look up registrations by trigger type | | | TrustStore.isTrusted() (for cross-repo events) | | | matchAllWorkflows() against registered workflows | | |-- Else (no RegistrationIndex): | | | TrustStore.isTrusted() (for cross-routing-key events) | | | matchAllWorkflows() against in-memory lock file subscriptions | | |-- For each match: onEventMatched(event, lockFile, matchedWorkflows) | | | |-- On success: markProcessed (commits processed=true, clears lease) | |-- On failure (any onEventMatched throws): | |-- If attempts >= maxDispatchAttempts: markDlq('exhausted_retries') | |-- Else: recordDispatchFailure (sets next_retry_at via exponential | backoff with full jitter; clears lease) | v Job dispatch to agents (standard pipeline) ``` ### At-least-once delivery + DLQ Two invariants keep events from being silently lost: - **Cron-fire atomicity:** `tryClaimFire` (advances `cron_last_fired`) and the event-row INSERT + `pg_notify` execute inside the same database transaction. If the leader process is killed between the two writes, the transaction rolls back and no `last_fired_at` advance leaks. The next tick re-evaluates and fires cleanly. - **Dispatch retries:** the lease pattern (`tryLeaseForProcessing`) marks an event as in-flight without committing it as processed. When a handler throws, the lease wrapper records the failure, schedules a retry, and on the leader's retry-scanner tick the event is re-published via `pg_notify`. After `maxDispatchAttempts` (default 5) the event lands in the DLQ (`dlq_at` set, `dlq_reason='exhausted_retries'`) and is surfaced via Prometheus (`kici_orch_event_dlq_*`), Grafana (`event-delivery` dashboard), and the kici-admin CLI (`kici-admin event-dlq {list,count,retry,discard}`). - **Crash detection:** when a node crashes mid-dispatch, its lease ages out after `leaseDurationMs` (default 60 s). The leader's `EventRetryScanner` releases the expired lease and re-publishes `pg_notify` so a healthy node picks the event up. Each release increments `kici_orch_event_lease_expirations_total` — a steady > 0 rate is the visible signal that an orchestrator instance is dying mid-dispatch. ### System events The orchestrator auto-emits system events after execution completes: - **`workflow_complete`** -- emitted when all jobs in a workflow finish (carries workflow name, status, duration) - **`job_complete`** -- emitted when a single job finishes (carries workflow name, job name, status, duration) These events are stored in the same `kici_events` table and matched against `workflowComplete()` and `jobComplete()` triggers in the lock file. ### Event.emit WS protocol ``` Agent Orchestrator | | |-- event.emit ----------------->| | { jobId, requestId, | | eventName, payload, | | target? } | | |-- store event | |-- NOTIFY |<-- event.emit.response --------| | { requestId, deliveryId? } | | | ``` ### Registration extraction flow When code is pushed to the default branch, the orchestrator extracts registerable workflows from the lock file and stores them as registrations for cluster-wide event matching. Non-Git triggers (`kici_event`, `schedule`, `generic_webhook`, …) live there because they have no per-repo lock-file pipeline to fall back on; Git-provider triggers (`push`, `pr`, `tag`, …) are indexed too, so the cross-source dispatch path can resolve them by `(customer_id, repo_identifier)` when a generic webhook targets an externally-hosted repo. For same-source Git events the per-event lock-file pipeline remains the primary matcher — registration is an additive index. ``` Git Push to Default Branch ========================== GitHub Webhook -> Platform Relay -> Orchestrator Processor Processor (on default-branch push): |-- lockFileCache.get() (fetch/cache lock file by blob SHA) |-- extractRegisterableWorkflows(fullLockFile) | |-- For each workflow entry in lock file: | | Check if any trigger type is registerable | | (the RegisterableTriggerType enum — the non-Git | | set kici_event, workflow_complete, | | workflows_failed_batch, job_complete, | | generic_webhook, schedule, lifecycle, webhook, | | plus every Git-provider trigger: push, pr, tag, | | comment, review, review_comment, release, | | dispatch, create, delete, status, workflow_run, | | fork, star, watch) | | ... or the workflow has repo patterns (global workflow) | |-- Return array of registerable workflows | |-- globalWorkflowPolicy.isWorkflowRepoAllowed() (if policy configured) | |-- Filter out global workflows from repos not on the allow-list | |-- registrationStore.replaceAll(repoIdentifier, workflows, routingKey, credentials, { commitSha }) | |-- BEGIN TRANSACTION | |-- DELETE FROM workflow_registrations WHERE routing_key AND repo_identifier | |-- INSERT new registrations (with commit SHA for lock file pinning) | |-- COMMIT | |-- registrationStore.bumpVersion() | |-- UPDATE registry_versions SET version = version + 1 | |-- registrationIndex.refreshIfNeeded(newVersion) | |-- If local version != remote version: | | Load all registrations from DB | | Rebuild primary index (by customer:repo) | | Rebuild secondary index (by trigger type) | | Update local version | |-- cronScheduler.refreshCache() (defense-in-depth) | |-- eventRouter.emit('registration.updated', { repo, workflows }) ``` ### Cron schedule evaluation flow Cron schedules are evaluated periodically by the Raft leader only. ``` Cron Schedule Evaluation ======================== CronScheduler (runs every 30 seconds, Raft leader only): |-- registrationIndex.getCronSchedules() |-- For each schedule: | |-- new Cron(cronExpression, { timezone }) | |-- cron.previousRuns(1) -> most recent past scheduled time | |-- Check last-fired cache (prevent double-fire) | |-- If due and not recently fired: | |-- BEGIN TRANSACTION | | |-- cronStore.tryClaimFire(registrationId, previousRun, tx) | | | (atomic DB claim — prevents duplicate fires in | | | multi-orchestrator clusters via WHERE last_fired_at < | | | firedAt guard) | | |-- If claim successful: | | |-- eventRouter.emitInTx(__schedule_fire, tx) | | |-- EventStore.writeWith(tx) -> INSERT kici_events | | |-- pg_notify('kici_event_channel', id) on tx | |-- COMMIT (rollback discards both writes; pg_notify fires on commit) | |-- On commit: | |-- Update local last-fired cache | |-- EventRouter matches against registered workflows | |-- Matched workflows dispatched via standard pipeline ``` Recovery on leader election loads the `cron_last_fired` table into the last-fired cache and fires once per missed schedule. Because the claim and the event-row insert now share a transaction, a crash between the two no longer leaves `last_fired_at` advanced with no event row — the rollback discards both writes and the next tick fires cleanly. #### Timing characteristics - **Tick interval:** Hardcoded at 30 s (`evaluationIntervalMs` defaults to `30_000` in `CronScheduler` and is not exposed via orchestrator config or env vars). Changing it requires a code change. - **Fire jitter:** A schedule due at time `T` fires at the first tick `>= T`, i.e. `0–30 s` after the scheduled moment, never before. The event payload's `scheduledAt` carries the cron-computed time (not the dispatch time), so downstream consumers see the intended schedule. - **Per-tick concurrency:** All schedules are processed serially in a single `for` loop on the leader (`packages/orchestrator/src/cron/cron-scheduler.ts`, `evaluate()`). Each registration costs one in-memory cron computation plus two DB writes (`tryClaimFire` upsert + `eventStore.write` + `pg_notify`). Throughput is therefore bounded by sequential DB write latency: at ~5–15 ms per registration, 50 schedules firing in the same tick complete in well under a second between the first and last fire. - **Recovery semantics:** On leader election, `recoverMissedSchedules()` calls `cron.previousRuns(1)` per schedule -- it fires at most one event per schedule regardless of how long the cluster was leaderless. There is no backfill for multiple missed scheduled instants. - **Multi-node deduplication:** During cluster startup multiple nodes may transiently self-elect (dormant mode). The atomic `tryClaimFire` upsert with a `last_fired_at < firedAt` `WHERE` guard ensures only one node's emit succeeds; losing nodes update their local cache and skip emit. - **Sub-minute crons:** Supported but bounded by the 30 s tick. `* * * * *` fires roughly once per minute with up to 30 s of drift; sub-30-second cadences are not achievable without lowering the interval in code. ## Generic webhook flow Generic webhooks from non-GitHub sources follow a parallel ingestion path. The webhook can arrive directly at the orchestrator or be relayed through the Platform. ``` External Service (ArgoCD, Jenkins, Grafana, etc.) | v POST /webhook/:orgId/generic/:sourceId | +--> Platform path: | |-- Resolve source by routing key generic:: | |-- Relay via WebSocket to orchestrator (see internal/platform/data-flows.md) | v +--> Orchestrator path (direct or via Platform relay): |-- GenericSourceManager.getByOrgAndName(orgId, sourceId) |-- Payload size check (per-source maxPayloadBytes) |-- Rate limit check (per-source rateLimitRpm) |-- Verify signature (HMAC-SHA256, bearer token, IP allowlist, or none) |-- Deduplication check (idempotency key within dedup window) |-- GenericWebhookNormalizer.normalizeEvent() -> SimulatedEvent |-- Match against lock file triggers (genericWebhook type) |-- Dispatch matched jobs to agents ``` ### Generic vs GitHub webhook differences | Aspect | GitHub Webhooks | Generic Webhooks | | -------------- | --------------------------- | ----------------------------------------------- | | Signature | HMAC-SHA256 (always) | Configurable (HMAC, bearer, IP, none) | | Event type | X-GitHub-Event header | Configurable header or payload field | | Delivery ID | X-GitHub-Delivery header | Configurable header or auto-generated UUID | | Lock file | Fetched from repo | Cached from lock file subscription | | Git operations | Clone, fetch, changed files | None (optional -- non-repo workflows supported) | ## Database topology The orchestrator owns its own PostgreSQL database, with the authoritative `execution_runs`, `execution_jobs`, `execution_steps`, `dispatch_queue`, `dedup_cache`, `workflow_registrations`, `contexts` / `scoped_secrets` / `context_bindings`, `agent_tokens`, `cluster_meta`, and related tables. Each orchestrator deployment uses its own `KICI_DATABASE_URL`; database users are scoped per service. ## Execution reporting flow After job execution, results flow back through the tiers: ``` Agent Orchestrator Platform GitHub | | | | |-- job.status ----------->| | | | (completed/failed) | | | | |-- execution.status ->| | | | (run metadata) |-- upsert | | | | execution_runs | | |-- job.status.forward>| | | | (job metadata) |-- upsert | | | | execution_jobs | | |-- GitHub Checks API ---------------------->| | | (check run update) | | | | | | ``` The orchestrator updates: 1. **GitHub Check Runs** via the Checks API (conclusion, summary, duration) 2. **Execution runs** in the orchestrator's own database (authoritative source) 3. **Platform execution status** via WebSocket (`execution.status` and `job.status.forward` messages, which the Platform upserts into its own projection tables) ## Re-run and cancel flows The dashboard enables users to re-run completed workflows and cancel running workflows. Both flows use a REST-over-WS proxy pattern: the Platform receives a REST request from the dashboard, forwards it to the orchestrator via WebSocket, and returns the orchestrator's response. ### Re-run flow ``` Dashboard Platform Orchestrator | | | |-- POST /orgs/:id/runs/ ->| | | :runId/rerun (auth) |-- Cooldown check | | | (last_rerun_at < 5s ago?) | | | | | |-- run.rerun.request (WS) --->| | | { runId, triggeredBy } | | | |-- Load original run from DB | | |-- Read webhook payload from storage | | |-- Re-fetch lock file at original SHA | | |-- Dispatch new jobs via Dispatcher | | |-- Record execution with parent_run_id + original_run_id | | |-- execution.status (WS, via callback) | | | { parentRunId, originalRunId, triggeredBy } | |<- run.rerun.response (WS) ---| | | { newRunId } | | | | |<- 200 { newRunId } ------| | | |-- UPDATE last_rerun_at | |-- Navigate to new run | | | | | ``` Key design points: - **Cooldown enforcement:** The Platform enforces a 5-second cooldown per original run via the `last_rerun_at` column. Rapid re-run attempts receive 429 Too Many Requests. - **Payload reuse:** The orchestrator reads the original webhook payload from filesystem/object storage and stores a copy for the new run (enabling re-run of re-runs). - **Lock file at original SHA:** The lock file is re-fetched at the original commit SHA, ensuring the re-run uses the same workflow definition. - **Lineage tracking:** The new run has `parent_run_id` pointing to the immediate parent run, `original_run_id` pointing to the root ancestor run (for chain traversal), and `triggered_by` recording the user identity. - **No trigger matching:** Re-runs skip deduplication, normalization, and trigger matching. They go directly from lock file parse to job dispatch. ### Cancel flow ``` Dashboard Platform Orchestrator Agent(s) | | | | |-- POST /orgs/:id/runs/ ->| | | | :runId/cancel (auth) | | | | |-- run.cancel.request (WS) -->| | | | { runId, cancelledBy } | | | | |-- Find active jobs | | | | from dispatch queue | | |-- job.cancel (WS) ->| | | | (for each agent) |-- Abort step | | | |-- Cleanup | |<- run.cancel.response (WS) --| | | | { cancelledJobs: N } | | | | |<- job.status -------| |<- 200 { cancelledJobs } -| | (cancelled) | | |-- UPDATE cancelled_by | | | | | | ``` The cancel flow is asynchronous: the orchestrator sends `job.cancel` to agents and immediately responds with the count. Agents asynchronously abort their current step, clean up, and report `job.status: cancelled` back to the orchestrator. ### Payload storage flow Webhook payloads are stored during initial processing and retrieved later for re-runs and the payload viewer. ``` Webhook arrives Payload retrieved | | v v processWebhook() GET /orgs/:id/runs/:runId/payload | | v v logStorage.append( Platform -> dashboard.payload (WS) executions/{runId}/ | webhook-payload.json, v JSON.stringify(payload) Orchestrator -> logStorage.read( ) executions/{runId}/ | webhook-payload.json v ) Filesystem or object storage | v dashboard.payload.response (WS) { payload: {...} } ``` ### Event-log payload streaming flow The dashboard's event-log detail panel reads webhook bodies through a chunked transport so the dashboard can render progress as bytes arrive. The orchestrator slices the payload into 64 KiB chunks and streams them up to the browser. ### Lineage query The lineage endpoint (`GET /orgs/:customerId/runs/:runId/reruns`) returns all runs with `parent_run_id` matching the given run ID. ## Trace ID propagation Every webhook event is assigned a trace ID (`requestId`) at ingestion. A second ID (`runId`) is added at dispatch time. Both propagate through the three tiers via WebSocket protocol messages and are automatically injected into every log line using AsyncLocalStorage. ``` GitHub Platform Orchestrator Agent | | | | |-- webhook -->| | | | |-- generate requestId | | | |-- requestContext.run() | | | | (requestId) | | | | | | | |-- webhook.relay (WS) ->| | | | { ..., requestId } | | | | |-- requestContext.run() | | | | (requestId) | | | | | | | |-- generate runId | | | |-- enrichRequestContext | | | | ({ runId }) | | | | | | | |-- job.dispatch (WS) ->| | | | { ..., requestId } | | | | |-- requestContext.run() | | | | (requestId, runId, | | | | jobId) | | | | | | | |-- log: "Run: X | Trace: Y" | | | |-- execute steps | | | | ``` ### How it works 1. **Platform ingestion:** The webhook handler generates a `requestId` (UUID) and wraps the entire request in `requestContext.run()`. All log lines within this async scope automatically include `requestId`. 2. **WebSocket relay:** The `requestId` is included in the `webhook.relay` message sent to the orchestrator. For cross-instance relay (via Valkey pub/sub), the `requestId` is serialized in the notification payload. 3. **Orchestrator processing:** The orchestrator wraps webhook processing in `requestContext.run()` using the `requestId` from the relay message (falling back to a new UUID for backward compatibility). When a `runId` is generated for job dispatch, it is enriched into the existing context via `enrichRequestContext()`. 4. **Job dispatch:** Both `requestId` and `runId` are included in the `job.dispatch` WebSocket message to the agent. 5. **Agent execution:** The agent wraps each `onJobDispatch` callback in `requestContext.run()` with `requestId`, `runId`, and `jobId`. A trace header is printed once at job start. All subsequent log lines carry all trace fields automatically. 6. **Check run summaries:** GitHub Check Run updates include `Trace: | Run: ` in the summary text, giving operators a direct link from GitHub UI to Loki queries. ### Implementation Trace propagation uses Node.js `AsyncLocalStorage` from `@kici-dev/shared`. A logger format reads the current context and injects fields into every JSON log line -- no changes needed at individual call sites. Tier identification is handled at the infrastructure level: the `service` Loki label (set by Grafana Alloy from the systemd unit / log source) identifies which service produced the log (`platform`, `orchestrator`, `agent`, etc.). For agent logs forwarded through the orchestrator's stdout, the parsed JSON also carries an inner `service: 'agent'` field — query both with `{service="orchestrator"} | json | service="agent"` to disambiguate. ## Output chaining data flow Output chaining allows steps to consume outputs from preceding steps (within a job) and jobs to consume outputs from preceding jobs (across jobs). The data flows through several phases. ### Definition time When workflow code runs at definition time (`step()`, `job()` calls): - `step()` creates an `OutputProxy` via `createStepOutputProxy(stepName)` and attaches it as `.result` - `job()` creates an `OutputProxy` (the job's inferred output shape — nested by step name for a multi-step job, flat for the `run:` shorthand) via `createJobOutputProxy(jobName)` and attaches it as `.result`, so cross-job reads type-check - The proxy is an ES6 `Proxy` object that defers all property access to a module-global `OutputsMap` - No outputs exist yet -- accessing `.result.field` before execution throws "has not produced outputs yet" ### Compile time The compiler processes the workflow definition: - Unnamed steps (bare functions and id-less `step()` calls) receive counter IDs: `step-1`, `step-2`, etc. - Unnamed jobs (id-less `job()` calls with UUID names) receive counter IDs: `job-1`, `job-2`, etc. - The lock file records `hasOutputs: true` for steps with Zod output schemas - Step counters are scoped per job; job counters are scoped per workflow ### Execution time (local test runner) When `kici run --local` runs a workflow: 1. **SDK module resolution:** The runner resolves `setStepOutputsMap` / `setJobOutputsMap` from the same `@kici-dev/sdk` module instance that the workflow uses (ensures the proxy reads from the same map) 2. **Map injection:** Fresh `OutputsMap` and `StepRefMap` are created and injected via `setStepOutputsMap()` / `setStepRefMap()` before each job 3. **Step execution:** Each step runs sequentially. If the step returns a value, it is stored in the `OutputsMap` keyed by step name 4. **Bare function normalization:** Bare functions in the steps array are assigned counter names and registered in the `StepRefMap` (maps function reference to step name) 5. **Proxy resolution:** When a subsequent step accesses `stepRef.result.field`, the proxy reads from the `OutputsMap` 6. **ctx.outputsOf():** Resolves step outputs by reference (Step object or bare function). For bare functions, looks up the step name in the `StepRefMap` ### Cross-job output aggregation After each job completes in the local test runner: 1. Step outputs from the completed job are aggregated into the `jobOutputsMap` 2. **Multi-step jobs:** Outputs are nested under step names: `{ stepName: { field: value }, ... }` 3. **Single-step jobs (run shorthand):** Outputs are flattened directly: `{ field: value }` (no step-name nesting) 4. The `jobOutputsMap` is injected via `setJobOutputsMap()`, enabling `jobRef.result.stepName.field` or `jobRef.result.field` access ### IPC transport (agent sandbox) In the agent sandbox (remote pipeline execution): 1. Step return values are captured and included in `step.complete` IPC messages (optional `outputs` field) 2. The agent aggregates step outputs and includes them in the `job.complete` IPC message 3. **Within-job chaining:** The sandbox populates the `OutputsMap` as steps complete, so `.result` and `ctx.outputsOf()` resolve correctly within a single job 4. **Cross-job chaining:** The orchestrator collects plain outputs from completed upstream jobs at dispatch time (querying the DB for jobs listed in `needs`), then passes them as `upstreamJobOutputs` in the `job.dispatch` message. The sandbox receives this map and populates the `jobOutputsMap` via `setJobOutputsMap()`, enabling `ctx.jobOutputs()` and `jobRef.result` access across job boundaries. Secret outputs follow a separate encrypted path via `SecretOutputStore`. #### Within-job output flow ``` Step A completes Step B accesses A.result | | v v Return value Proxy.get('field') | | v v OutputsMap.set('A', val) OutputsMap.get('A') | | v v Stored in shared map Returns val.field ``` #### Cross-job output flow ``` Job A completes Orchestrator dispatches Job B | | v v Outputs stored in DB Query upstream job outputs (needs) | v job.dispatch includes upstreamJobOutputs | v Sandbox populates jobOutputsMap | v ctx.jobOutputs('A') resolves ``` ## Browser protocol (Platform to dashboard) The Platform tier exposes a `/ws/browser` WebSocket endpoint for dashboard clients (auth, log subscription / streaming / gaps, run / job / step status updates, `run.event` / `job.context` for the Summary tab). ## See also - [Architecture overview](https://docs.kici.dev/architecture/overview/) -- three-tier model and component responsibilities - [Protocol messages](https://docs.kici.dev/architecture/protocol-messages/) -- WebSocket message schemas - [Event system internals](https://docs.kici.dev/architecture/webhooks/event-system/) -- event router, registration model, cron scheduler - [Execution status vocabulary](https://docs.kici.dev/architecture/execution/state-machine/) -- run, job, and step status vocabularies and terminal states - [Webhook delivery](https://docs.kici.dev/architecture/webhooks/webhook-delivery/) -- detailed webhook processing pipeline - [Operator: dependency caching](https://docs.kici.dev/operator/dependency-caching/) -- configuration guide - [Operator: monitoring & tracing](https://docs.kici.dev/operator/observability/monitoring/) -- trace fields and Loki queries - [Operator: event routing & generic webhooks](https://docs.kici.dev/operator/event-routing/) -- generic source setup and trust management - [SDK reference: output chaining](https://docs.kici.dev/user/sdk/core/#output-chaining) -- user-facing output chaining API --- ## Architecture overview Source: https://docs.kici.dev/architecture/overview/ KiCI uses a three-tier relay model that separates webhook routing from code execution. Customer code never leaves customer infrastructure -- the Platform tier handles only webhook verification and routing, while the orchestrator and agent tiers run on customer-managed servers. ## Three-tier relay model The system is organized into three deployment tiers connected by WebSocket channels: ```mermaid flowchart LR GH["GitHub"] PLATFORM["Platform\nWebhook router"] ORCH_A["Orchestrator A\nExecution brain"] ORCH_B["Orchestrator B\nExecution brain"] AGENT_A["Agent\n(x64)"] AGENT_B["Agent\n(arm64)"] GH -- "HTTP\n(webhooks)" --> PLATFORM GH -. "HTTP direct webhook\n(hybrid / observed / independent)" .-> ORCH_A PLATFORM <-- "WebSocket\n(relay + telemetry)" --> ORCH_A PLATFORM <-- "WebSocket\n(relay + telemetry)" --> ORCH_B ORCH_A <-- "WebSocket P2P\n(reroute + progress\n+ Raft)" --> ORCH_B ORCH_A <-- "WebSocket\n(dispatch + status)" --> AGENT_A ORCH_B <-- "WebSocket\n(dispatch + status)" --> AGENT_B ORCH_A -- "GitHub API\n(lock file, checks)" --> GH AGENT_A -- "git clone" --> GH AGENT_B -- "git clone" --> GH ``` **Why three tiers?** Trust boundaries. The Platform relay never sees customer code -- it only verifies webhook signatures and forwards payloads. The orchestrator matches triggers against the lock file without cloning repositories. Only the agent, running on customer infrastructure, clones code and executes steps. This model also supports pointing webhooks **directly** at the orchestrator, bypassing the Platform relay. In **hybrid** mode the orchestrator keeps its Platform connection for the dashboard and telemetry while GitHub delivers events straight to the orchestrator's ingress — so a Platform outage never drops a build trigger. **Observed** mode drops the relay leg entirely: webhooks reach only the orchestrator's own ingress (no payload ever transits KiCI) while the Platform connection stays up for the hosted dashboard, and the orchestrator's sources register as observe-only — recorded and dashboard-visible, but excluded from every relay-candidate lookup. In a fully **independent** deployment the orchestrator and agent run on customer infrastructure with no Platform at all, receiving webhooks directly. For exactly which capabilities the hosted Platform provides in each case, see [What requires the hosted Platform](https://docs.kici.dev/operator/orchestrator/platform-capabilities/). ## Component responsibilities ### Platform The Platform is KiCI's hosted, multi-tenant control plane. It provides the hosted dashboard (run listing, run detail, live log streaming, settings), identity and authentication (OIDC, personal access tokens, API keys, JWTs), multi-tenant organization / team / role-based access management, billing, and webhook ingestion -- verifying inbound signatures (HMAC-SHA256, timing-safe) and relaying payloads to the correct orchestrator over WebSocket. It aggregates execution telemetry and status forwarded by orchestrators, registers sources, and matchmakes peers for clustering. The Platform never processes, stores, or executes customer code, and never sees customer secrets. It routes webhook payloads and aggregates execution status; the code itself only ever lives on the customer's orchestrator and agent tiers. In the execution path the Platform is deliberately thin -- it does not run jobs -- but functionally it is a full platform, not merely a relay. The hosted Platform is EU-sovereign. ### Orchestrator (`@kici-dev/orchestrator`) The orchestrator is the execution brain. It decides what to run and dispatches work to agents. - **Trigger matching** -- Evaluates lock file triggers against webhook payloads to determine which jobs to run. Uses branch, path, and event matching via picomatch. - **Lock file caching** -- Fetches `kici.lock.json` via the configured source's fetcher (GitHub API, universal-git clone for generic webhook sources backed by a git URL, or the local filesystem for `file://` sources). An LRU cache wraps the per-provider fetcher, keyed by `{provider}:{repo}:{ref}` so cross-provider fallback resolutions stay isolated. - **Agent registry** -- Tracks connected agents with label-based routing for job dispatch. - **Job queue** -- PostgreSQL-backed FIFO queue for reliable dispatch. - **Webhook pipeline** -- Dedup, event mapping, lock file fetch, trigger matching, and job dispatch in a single pipeline. - **Multi-orchestrator clustering** -- Optional peer-to-peer coordination via direct WebSocket connections. Enables cross-architecture job routing (e.g., x64 coordinator reroutes arm64 jobs to a peer), high availability, and dedicated coordinator topologies. Uses Raft consensus for leader election (orphan recovery). See [Multi-Orchestrator Architecture](https://docs.kici.dev/architecture/clustering/multi-orchestrator/). - **Auto-scaler** -- Optional pluggable module for ephemeral agent provisioning. Four backends are configurable: containers (Docker/Podman), bare-metal processes, Firecracker microVMs, and the event backend, which performs no local compute -- it emits reserved scale-up / scale-down events that a customer-authored provisioning workflow consumes to boot and tear down a cloud instance. Spawns agents on demand when no matching agent is connected, with label-based routing, two-level capacity limits (global + per-backend), warm pools, YAML configuration (`scalers.d/` directory support), and SIGHUP reload. Disabled by default -- orchestrator works without it. - **Independent database** -- Has its own PostgreSQL database separate from the Platform. Stores execution runs/jobs/steps, dispatch queue, webhook secrets, dedup cache, and scaler state. The orchestrator's `execution_runs` and `execution_jobs` are the authoritative source of truth. The Platform receives execution status updates via WebSocket messages (`execution.status`, `job.status.forward`). > Source: `packages/orchestrator/src/pipeline/processor.ts` (webhook pipeline), `packages/orchestrator/src/cluster/` (P2P coordination), `packages/orchestrator/src/scaler/` (auto-scaler module), `packages/orchestrator/src/server.ts` (Platform/hybrid entry point) ### Agent (`@kici-dev/agent`) The agent is the execution worker. It runs on customer infrastructure and has full access to customer code. - **Repository cloning** -- Clones the target repo with token-based auth (token in HTTP headers, not URLs, to prevent leakage). - **Git credential helper** -- Registers a credential helper for the job's git operations. Every network operation asks the orchestrator's broker for a credential, so a token is minted seconds before use rather than held for the life of the job. Write access is opt-in and time-boxed: `ctx.repo.withWrite(...)` adds a repository-scoped grant for the duration of its callback and revokes it afterwards, with a TTL backstop. - **Step execution** -- Runs steps in declaration order with full `StepContext` (zx shell, logger, environment, workflow/job metadata). Steps wrapped in a `parallel()` group run concurrently behind a `maxParallel` window, and each child reports as its own observable step with its own logs, status, timing, and retry. - **Execution sandboxes** -- Runs the workflow runner as a separate child process with a sanitized environment, in one of three sandboxes: bare metal (process fork, with optional bubblewrap namespace isolation), a container runtime (the whole job lifecycle runs inside a disposable container), or inside a Firecracker microVM. Agent-internal credentials never reach customer workflow code. - **Log streaming** -- Chunked log streaming back to the orchestrator with configurable size limits. - **Dependency caching** -- Packs, uploads, and restores installed workflow dependencies so repeat runs skip the install step. - **Graceful shutdown** -- SIGTERM with 10s grace period, SIGUSR1 for drain mode. > Source: `packages/agent/src/execution/job-runner.ts` (job lifecycle), `packages/agent/src/execution/sandbox/` (execution sandboxes and the parallel step scheduler), `packages/agent/src/server.ts` (entry point) ## Supporting packages ### `@kici-dev/engine` Shared business logic used by all three tiers. Single source of truth for cross-tier concerns. Has no internal `@kici-dev/*` dependencies -- only a handful of third-party libraries. - Protocol message schemas (Zod-based, direction-specific unions including dashboard REST-over-WS, browser live streaming, the test-relay control plane, log pull, run events, peer-to-peer, cluster join, and source registration) - Provider interfaces (WebhookNormalizer, LockFileFetcher, ChangedFilesFetcher, FileContentsFetcher, CloneTokenProvider, RepoUrlBuilder, CheckStatusPoster), plus the deprecated `ContributorResolver` the pipeline no longer calls - Git credential vocabulary (forge names plus the credential reference, grant, request, and result shapes the SDK declares and the orchestrator's broker resolves) and the agent→orchestrator relay protocol its credential helper calls. See [Git credentials](https://docs.kici.dev/user/patterns/git-credentials/) - Trigger matching engine (branch, path, event evaluation) - Content-requirement matcher (the declarative `requires` filter -- pure data describing a query over the bytes of one source file at the event's ref, interpreted by the orchestrator via the `FileContentsFetcher` so no author code runs there) and the shared text-match vocabulary (`contains` / `notContains` / `matches` / `notMatches`) it shares with the commit-message trigger filter - Dispatch inputs (input descriptors, extraction from the trigger event, and coercion to typed values) - Matrix expansion and fanout (combination expansion with include/exclude, job-name suffix formatting, and materialization of one matrix or multi-host job into N dispatchable children) - Execution status vocabulary (run/job/step status enums + terminal-state sets; lifecycle owned by the orchestrator's execution tracker) and its presentation layer (the total precedence order that decides which status wins a roll-up, legacy-spelling resolution, and the per-status failure classification every consumer asks about) - Job-kind discriminator, alongside the status enums. It separates a `standard` job running steps from an invoke `gate` and from the `proxy` job that mirrors a summoned run - Check mode (the idempotent run modes `apply` / `check` / `check-fail-on-drift` and the per-step outcome vocabulary) - Webhook signature verification (HMAC-SHA256, timing-safe) - WebSocket close codes (unified across all tiers) - WebSocket rate limiting (WsRateLimiter) - Environment allowlist (safe env var filtering) - Secrets management (secret context resolution) - Context model (scoped secrets, ordered context merge, protection gates) - Approval requirements (normalized approver clauses shared by the orchestrator gate, the resolver, the held-run store, and the agent step round-trip) - Build provenance (in-toto statement schema, DSSE envelope, attestation bundle, verification) - Artifact name contract (the shared filesystem/URL-safe name schema the orchestrator, agent, and SDK all validate against) - Developer MCP tool schemas (argument schemas for the AI-agent tool surface) and the untrusted-content fence that wraps every repository- or contributor-supplied value an agent reads in a per-response random nonce, so log lines and error text cannot be read as instructions - Developer-operations contract (one row per workflow-developer operation declaring which entrypoints expose it -- the shared REST API behind the web UI and the `kici` CLI, the AI-agent tool surface, and a curated UI flag -- asserted against each real surface by congruence tests) - Label utilities (platform label derivation, runsOn normalization, `kici:*` set-only reserved namespace, role labels) - Host inventory (the canonical queryable host-roster schema shared by the orchestrator's roster store, the agent-facing inventory API, and the SDK's `ctx.kici.inventory`) - Audit policy and retention (per-action access-log sampling, warm-retention windows for cold-store eligibility, federated activity row schema) - Scaler backend type enum (`container`, `bare-metal`, `firecracker`, `kubernetes`, `event`; the orchestrator config rejects `kubernetes`) and the reserved `kici.` event-name prefix that keeps a user step from forging a system event - Job resource vocabulary (the requests/limits shape the SDK accepts, the compiler validates and emits, the orchestrator uses for capacity math and kernel-side enforcement, and the dashboard displays) - Registration trigger type enum (registerable trigger discriminator) - Sandbox capability set (the Linux capability names a container sandbox may add or drop, shared by the SDK validator, the compiler, and the dispatch resolver) - Plan tier vocabulary (the hosted plan tiers and the purchasable subset, shared by the Platform and the browser dashboard) - Infrastructure alert vocabulary (the diagnostics alert types and severities the Platform mints and the dashboard and `kici` CLI render) - Metric catalog (the generated Prometheus metric inventory, its naming policy, and metric-kind compatibility checks) - Bundler config (the shared workflow-bundle configuration factory on the barrel; the agent runtime uses the `@kici-dev/core/ts-loader-hook` to transform TypeScript on import, so no runtime path bundles a workflow) > Source: `packages/engine/src/` ### `@kici-dev/sdk` User-facing SDK for defining workflows in TypeScript. Provides factory functions (`workflow()`, `job()`, `step()`), trigger builders (`pr()`, `push()`), rules (`rule()`, `skip()`), matrix utilities, and DAG validation. > Source: `packages/sdk/src/` ### `@kici-dev/compiler` CLI tooling for workflow authors. Compiles `.kici/workflows/*.ts` to `.kici/kici.lock.json`, provides watch mode, local test execution, project initialization, and pre-commit hook integration. It also runs the **local dev plane** -- an on-demand, fully local execution stack (embedded PostgreSQL, an orchestrator process, and a bare-metal-scaled agent) that lets an author run a workflow end-to-end on their own machine. That is why the compiler depends on `@kici-dev/orchestrator` and `@kici-dev/agent`: it resolves and spawns their built entry points rather than reimplementing them. See [Local dev plane](https://docs.kici.dev/operator/orchestrator/local-dev-plane/). > Source: `packages/compiler/src/` (`local-plane/` for the local dev plane) ### `@kici-dev/core` Light shared utilities with no server-side dependencies. It provides JSON-structured logging, error helpers, async-local-storage request context, and human-readable formatting (`formatBytes`/`formatDuration`/`formatUptime`). It also provides cryptographic helpers (`sha256`/`sha256File`/`deriveSharedSecret` plus symmetric encrypt/decrypt), retry-backoff computation, and the shared diagnostics-result contract. The rest of its surface ships as subpath entry points: the temp-directory allocator and its garbage collector, package-manager detection, CI-environment detection, and the idempotent-step runner (the check / confirm / apply primitive behind idempotent steps). One further subpath holds the `.kici/` source digest: the single content-hash definition the compiler writes into the lock file and the agent recomputes as its drift gate. Finally it supplies zx initialization (`initZx()`) and the TypeScript loader hook that transforms TypeScript on import. It is the dependency-light core that the SDK, compiler, and `kici` CLI consume directly so they stay free of heavier server-only dependencies. `@kici-dev/shared` re-exports it, so existing `@kici-dev/shared` import paths keep working. > Source: `packages/core/src/` ### `@kici-dev/shared` Shared utilities used across packages, including everything from `@kici-dev/core` (re-exported) plus server-side helpers. Provides `initZx()` for zx initialization, `createLogger()` for JSON-structured logging with TTY-aware formatting, `createPool()`/`createDb()` for typed PostgreSQL connections, `createMetricsRoutes()`/`createHealthRoutes()` for HTTP route factories (Prometheus metrics and health endpoints), `RingBuffer` for bounded collections, `requestContext`/`getRequestContext()`/`enrichRequestContext()` for async local storage request context, `getReconnectDelay()` for exponential backoff, `formatBytes`/`formatDuration`/`formatUptime` for human-readable formatting, `sha256`/`sha256File`/`deriveSharedSecret` for cryptographic utilities, `initTelemetry`/`createMeter` for OpenTelemetry integration, and `setupGracefulShutdown` for coordinated service shutdown with ordered steps. > Source: `packages/shared/src/` ### Dashboard Web UI for KiCI. A browser single-page application that provides the operator dashboard with execution run listing, run detail views, real-time log streaming, settings management, and keyboard shortcut support. Authenticates via OIDC against the identity provider and communicates with the Platform REST-over-WebSocket API. ### `kici` (wrapper) Unscoped wrapper package that provides the `kici` CLI command. Re-exports `@kici-dev/compiler/cli` so users can install `kici` globally or use it via `npx kici`. > Source: `packages/kici/` ### `kici-admin` (admin CLI wrapper) Unscoped wrapper package that ships two binaries: `kici-admin`, which re-exports `@kici-dev/orchestrator/cli` for orchestrator administration tasks, and `kici-agent`, which re-exports `@kici-dev/agent/server` to run an agent. It therefore depends on both `@kici-dev/orchestrator` and `@kici-dev/agent` (the `KICIADMIN → AGENT` edge in the graph below). > Source: `packages/kici-admin/` ## Package dependency graph The following diagram shows how `@kici` packages depend on each other. Solid arrows are direct dependencies; dashed arrows are peer or dev dependencies (labeled). ```mermaid flowchart TD CORE["@kici-dev/core"] SDK["@kici-dev/sdk"] COMPILER["@kici-dev/compiler"] SHARED["@kici-dev/shared"] ENGINE["@kici-dev/engine"] PLATFORM["Platform"] ORCH["@kici-dev/orchestrator"] AGENT["@kici-dev/agent"] DASH["Dashboard"] DASH --> ENGINE DASH -.->|dev| PLATFORM SHARED --> CORE SHARED --> ENGINE SDK --> ENGINE SDK --> CORE COMPILER --> ENGINE COMPILER --> CORE COMPILER --> ORCH COMPILER --> AGENT COMPILER -.->|peer| SDK PLATFORM --> ENGINE PLATFORM --> SHARED ORCH --> ENGINE ORCH --> SHARED ORCH -.->|dev| AGENT AGENT --> ENGINE AGENT --> SDK AGENT --> SHARED AGENT --> CORE KICI["kici (wrapper)"] KICI --> COMPILER KICI --> CORE KICIADMIN["kici-admin (admin CLI)"] KICIADMIN --> ORCH KICIADMIN --> AGENT ``` **Leaf packages** (no `@kici` dependencies): `@kici-dev/core` and `@kici-dev/engine`. These can be tested and built independently. `@kici-dev/shared` builds on `@kici-dev/core` (which it re-exports) and on `@kici-dev/engine` for shared vocabularies. The dashboard depends on `@kici-dev/engine` for shared types (protocol schemas, execution status enums) and imports the Platform's API type definitions as a dev dependency, but communicates with backend services at runtime via HTTP/WebSocket, not at compile time. **Runtime tiers** (Platform, orchestrator, agent) all depend on `@kici-dev/engine` for shared business logic and `@kici-dev/shared` for utilities. Only the agent depends on `@kici-dev/sdk` (it loads workflow definitions at runtime). The `COMPILER → ORCH` and `COMPILER → AGENT` edges exist solely for the local dev plane: the compiler spawns a local orchestrator and agent so an author can execute a workflow end-to-end without any deployed infrastructure. Nothing in the compile path itself reaches into either tier. ## Connection overview KiCI uses three WebSocket layers for real-time communication. ### Platform ↔ Orchestrator The orchestrator connects outbound to the Platform WebSocket endpoint. After authentication (API key validated via SHA-256 hash lookup), the connection is used for webhook relay, execution telemetry (events, status, logs), source registration, and peer discovery. Peer discovery is matchmaking only: the Platform pushes a `peer.update` membership list to every orchestrator sharing a routing key, and the orchestrators then connect to each other directly. Inter-orchestrator traffic such as `job.reroute` never transits the Platform. ### Orchestrator ↔ Orchestrator (P2P) When multiple orchestrators are deployed, they establish direct WebSocket connections to each other on the `/ws/peer` endpoint. Peers are discovered via the Platform matchmaker (Platform/hybrid modes) or static configuration (`KICI_CLUSTER_PEERS` env var, independent mode). Connections are authenticated with a mutual pre-shared key (PSK). Traffic includes agent inventory heartbeats, job rerouting, progress reporting, cancel propagation, and Raft leader election. These messages never transit the Platform tier. > See [Multi-Orchestrator Architecture](https://docs.kici.dev/architecture/clustering/multi-orchestrator/) for clustering details and [Protocol Messages](https://docs.kici.dev/architecture/protocol/dashboard/#orchestrator---orchestrator-messages-peer-to-peer) for message schemas. ### Orchestrator ↔ Agent The agent connects outbound to the orchestrator WebSocket endpoint. After registration (agent ID, labels, concurrency), the connection is used for job dispatch, status reporting, and log streaming. > See [Protocol Messages](https://docs.kici.dev/architecture/protocol-messages/) and [Webhook Delivery](https://docs.kici.dev/architecture/webhooks/webhook-delivery/) for detailed message flows and schemas. ## Authentication and multi-tenancy KiCI uses application-level tenant isolation. The Platform dashboard API accepts three authentication methods (PATs, API keys, JWTs) and enforces org membership on every `/api/v1/orgs/:customerId/*` request. ## See also - [Multi-Orchestrator Architecture](https://docs.kici.dev/architecture/clustering/multi-orchestrator/) -- P2P clustering, Raft consensus, job rerouting - [Execution status vocabulary](https://docs.kici.dev/architecture/execution/state-machine/) -- run, job, and step status vocabularies and the tracker that owns lifecycle state - [Protocol Messages](https://docs.kici.dev/architecture/protocol-messages/) -- WebSocket message schemas for all three layers - [Webhook Delivery](https://docs.kici.dev/architecture/webhooks/webhook-delivery/) -- end-to-end trace of a webhook through all three tiers ---