Test run architecture

This document describes the end-to-end data flow for remote test runs triggered by kici run remote, including the upload encryption scheme, overlay application, log following, and how test runs integrate with the existing production pipeline.

High-level data flow

Developer workstation            Orchestrator              Agent
       |                              |                      |
  1. kici run remote push-main        |                      |
       |                              |                      |
  2. Compile fixture                  |                      |
       |                              |                      |
  3. POST /uploads/init  ------------>|                      |
       |<---- { signedUrl,            |                      |
       |        uploadId,             |                      |
       |        publicKey } --------- |                      |
       |                              |                      |
  4. Create overlay tarball           |                      |
       |                              |                      |
  5. Encrypt tarball (ECDH)           |                      |
       |                              |                      |
  6. PUT signed URL (S3) -----> [Object Storage]             |
       |                              |                      |
  7. POST /test/trigger  ------------>|                      |
       |<---- { runs[] } ------------ |                      |
       |                              |                      |
  8. Begin polling logs + status      |                      |
       |                              |                      |
       |                         9. Trigger match            |
       |                              |                      |
       |                        10. Dispatch -----> job.dispatch
       |                              |              (with tarballUrl,
       |                              |               cliPublicKey,
       |                              |               orchestratorPrivateKey)
       |                              |                      |
       |                              |                11. Clone repo
       |                              |                      |
       |                              |                12. Download tarball
       |                              |                      |
       |                              |                13. Decrypt (ECDH)
       |                              |                      |
       |                              |                14. Verify checksums
       |                              |                      |
       |                              |                15. Apply overlay
       |                              |                      |
       |                              |                16. Execute steps
       |                              |                      |
       | GET /test/runs/:id/logs ---->|<---- log.chunk ------|
       | GET /test/runs/:id --------->|<---- step.status ----|
       |   (polled to completion)     |<---- job.status -----|
       |                              |                      |
  17. Show summary + exit code        |                      |

Request path

The diagram draws the CLI’s control calls as if they go straight to the orchestrator. On the default path they reach it through the hosted KiCI Platform. The CLI calls the Platform’s /api/v1/orgs/:customerId/test/... endpoints with the developer’s personal access token. The Platform relays each call to the orchestrator over the orchestrator’s WebSocket connection as a test.relay.* message: test.relay.uploads.init, test.relay.trigger, test.relay.run.status, test.relay.run.logs, or test.relay.cancel. So the orchestrator does not have to be reachable from the developer’s network. Each relayed request carries the developer’s identity, which the orchestrator writes to its access_log.

In direct mode (kici connect, --orchestrator-url, or KICI_ORCHESTRATOR_TOKEN) the calls do go straight to the orchestrator: the CLI calls its /api/v1/test/* routes (whoami, uploads/init, trigger, runs/:runId, runs/:runId/logs, runs/:runId/cancel) with an orchestrator admin token. The routes run the same handlers as the relay, with the same payloads. The orchestrator authenticates the token itself, attributes the run to the token’s service account, and chooses the org and routing key.

The overlay tarball does not go through either control path. The CLI uploads it straight to object storage through the presigned URL that the upload-init call returns. The orchestrator signs it with KICI_STORAGE_UPLOAD_ENDPOINT, falling back to KICI_STORAGE_EXTERNAL_ENDPOINT, then KICI_STORAGE_ENDPOINT.

Upload encryption

Test run tarballs are encrypted using ephemeral X25519 ECDH key exchange with AES-256-GCM symmetric encryption. This ensures that uploaded content is protected in transit and at rest in object storage.

Key exchange flow

    CLI                     Orchestrator                Agent
     |                           |                        |
     |  POST /uploads/init       |                        |
     |-------------------------->|                        |
     |                           |                        |
     |  Generate orchestrator    |                        |
     |  ephemeral X25519 keypair |                        |
     |                           |                        |
     |  { publicKey, uploadId }  |                        |
     |<--------------------------|                        |
     |                           |                        |
     |  Generate CLI             |                        |
     |  ephemeral X25519 keypair |                        |
     |                           |                        |
     |  ECDH shared secret:      |                        |
     |  cliPrivate + orchPublic  |                        |
     |       = AES-256 key       |                        |
     |                           |                        |
     |  Encrypt tarball          |                        |
     |  AES-256-GCM              |                        |
     |                           |                        |
     |  Upload encrypted +       |                        |
     |  send cliPublicKey        |                        |
     |-------------------------->|                        |
     |                           |                        |
     |                           |  Dispatch job with:    |
     |                           |  tarballUrl,           |
     |                           |  cliPublicKey,         |
     |                           |  orchestratorPrivateKey|
     |                           |----------------------->|
     |                           |                        |
     |                           |  ECDH shared secret:   |
     |                           |  orchPrivate + cliPub  |
     |                           |       = AES-256 key    |
     |                           |                        |
     |                           |  Decrypt tarball       |
     |                           |  AES-256-GCM           |

Crypto details

Component	Algorithm
Key exchange	X25519 (Curve25519 ECDH)
Key derivation	HKDF-SHA256 with info `kici-upload-encryption`
Symmetric encryption	AES-256-GCM
Wire format	`[12-byte IV][16-byte auth tag][ciphertext]`
Key serialization	DER format (SPKI for public, PKCS8 for private)

The shared secret is derived via crypto.diffieHellman() and stretched through HKDF to produce a 32-byte AES key. Each upload uses fresh ephemeral keypairs — keys are never reused.

Security properties

Forward secrecy: Ephemeral keypairs mean compromising stored ciphertext later is useless without the keys (which are deleted after use)
Integrity: AES-256-GCM auth tag prevents tampering
No key reuse: Every upload generates fresh keypairs on both sides
No plaintext in storage: Object storage only ever holds encrypted data

Overlay application

The agent applies the developer’s local changes on top of a fresh git clone. This produces the exact same file state as the developer’s working tree.

Application flow

Agent clones repo at the SHA specified in the fixture
Agent downloads encrypted tarball from the URL provided in the job dispatch
Agent derives shared secret using orchestrator’s private key + CLI’s public key
Agent decrypts tarball using AES-256-GCM
Agent extracts tar.gz to a temporary directory
Agent reads manifest.json from .kici-overlay-tmp/ in the extracted files
Agent verifies SHA256 checksums of every extracted file against the manifest
Agent copies files from the extracted overlay to the clone directory
Agent deletes files listed in the manifest’s deletions array
Agent cleans up temporary files

Tarball structure

overlay.tar.gz
  .kici-overlay-tmp/
    manifest.json       # Checksums, deletions, HEAD SHA
  src/
    modified-file.ts    # Changed files at their repo-relative paths
    new-file.ts
  tests/
    added-test.ts

The CLI writes the manifest into the tarball from a temporary directory. It never writes into the working tree. .kici-overlay-tmp/ is reserved: the CLI never uploads a file at that path in the repository, because it would collide with the manifest.

Manifest format

{
  "sha": "abc123def456...",
  "deletions": ["src/removed-file.ts", "docs/old-guide.md"],
  "checksums": {
    "src/modified-file.ts": "sha256-hex-hash",
    "src/new-file.ts": "sha256-hex-hash",
    "tests/added-test.ts": "sha256-hex-hash"
  }
}

sha: The HEAD commit SHA the overlay is based on. The agent clones this exact commit.
deletions: Files the developer deleted locally. The agent removes these from the clone.
checksums: SHA256 hashes of each included file. The agent verifies these after extraction to detect corruption.

Following a test run

The CLI follows a test run by polling, over the same HTTP surface it used to trigger it — there is no streaming socket between the CLI and the orchestrator.

Poll loop

The trigger response returns runs: one { runId, workflowName, jobIds } per matched workflow, because each matched workflow runs as its own run. The CLI follows every run at once and passes only when every run passed. Ctrl-C cancels every run. The deprecated top-level runId names the first run, for a client that reads only it.
For each run, the CLI polls two endpoints in lockstep on a fixed interval: GET /api/v1/orgs/:customerId/test/runs/:runId/logs?cursor=<n>&logCursor=<opaque> for the next log chunk, and GET /api/v1/orgs/:customerId/test/runs/:runId for the status snapshot. Both address the cluster the trigger was relayed to.
Each log response carries lineJobNames, the job that wrote each line, so the CLI prints an error block per failed job with only that job’s output. Each log response also carries nextLogCursor, an opaque cursor that records how many lines of each log file were delivered. The CLI sends it on the next poll, so a chunk is never re-printed and never skipped, even when a step’s log file appears after a file that sorts later. The first poll sends an empty logCursor. When a response has no nextLogCursor, because the Platform or the orchestrator is older, the CLI uses the numeric line cursor nextCursor.
The run is finished only when the status is terminal and the log stream has drained (logs.done). A terminal status alone is not enough — the tail of the log can still be arriving.

Because the cursor lives in the CLI and every request is an ordinary authenticated HTTP call, a network blip needs no reconnection protocol: the next poll resumes from the same cursor. Ctrl-C cancels the run through the same client rather than just detaching.

Two grace behaviors keep the loop honest against a run the control plane has not observed yet: a 404 before the first successful read is retried until a visibility grace period elapses, and approval holds surfaced by a non-terminal tick are reported once each rather than on every poll.

Orchestrator-side broadcast

Inside the orchestrator, a per-run observer registry buffers the run’s observe.log / observe.step / observe.status / observe.complete messages with monotonic sequence numbers — up to 1000 messages per run, retained for five minutes after completion. The execution tracker and log writer publish into it only for runs marked as test runs. It is internal machinery: the buffer exists so a future subscriber can be backfilled from a sequence number, and nothing subscribes to it today.

Test runs vs production runs

Test runs share most of the production pipeline but differ in key ways:

Aspect	Production run	Test run
Trigger source	GitHub webhook	`POST /api/v1/orgs/:customerId/test/trigger`
Event normalization	Provider-specific normalizer	Synthetic event from fixture
Trigger matching	Lock file triggers	Same pipeline (or bypass with `--workflow`)
Dispatch core	Shared `dispatchMatchedWorkflow`	Same shared core (needs DAG, host fan-out, dynamic)
Repo state	Exact commit from webhook	Clone + overlay of local changes
Secret access	All contexts	Only `allowLocalExecution: true` contexts
Tracking	`execution_runs` table	Same table with `is_test_run = true`
Delivery ID	Provider-assigned	`test:` prefix + UUID
Log following	Dashboard / `kici runs` only	CLI polls logs + status to completion
`ctx.isTestRun`	`false`	`true`

One dispatch core, two adapters

There is a single dispatch core (dispatchMatchedWorkflow) and two thin adapters that feed it: the webhook adapter (real provider event) and the test adapter (processTestTrigger, for kici run). The webhook-only preamble — delivery dedup, Platform relay, provider normalization, source registration — runs in the caller, not inside the core, so the test adapter reaches the same core without it.

The test adapter:

Resolves the lock file (inline for a local repo with no remote, or the same provider-driven fetch the webhook uses).
Selects matched workflow decisions (normal trigger matching, or a direct --workflow bypass).
Enforces the allowLocalExecution environment gate and stores the fixture payload.
Mints one run id per matched workflow, as the webhook adapter does, then builds a dispatch context per matched workflow — a synthetic WebhookInfo, the test provenance fields, and a CLI-secret overlay that wins over orchestrator env secrets — and calls the shared core. A run keys its needs edges, pending contexts, placeholders, holds and log paths by job name, so two workflows that define a job of the same name must not share one. A caller that does not send acceptsMultipleRuns: true (a kici older than 0.21.0) follows only one run: for it, the adapter runs the matched workflows as one shared run when no job name collides, and refuses the trigger when one can.
Marks the execution as isTestRun for the orchestrator-side observer broadcast and secret gating.

Because the test adapter calls the same core, a kici run exercises the full dispatch behavior — needs-DAG scheduling, runsOnAll host fan-out, matrix and fan-out edge wiring, and deferred init/dynamic job dispatch — exactly as a webhook does. A multi-job needs workflow run via kici run honors the dependency DAG (a downstream job dispatches only after its upstream reaches a matching state), and a runsOnAll job fans out to one pinned execution per matching roster host.

Dispatch parity

A test-run job.dispatch carries the same execution-shaping fields as a production dispatch, all derived from the fixture’s simulated event: the normalized event envelope ({ type, action, targetBranch, sourceBranch, changedFiles, payload, … }), the resolved job env, the resolved environment name, and that environment’s variables. Dynamic functions evaluate against this envelope exactly as they do in production. A pure inline environment expression is evaluated at the orchestrator and its resolved name is subject to the bound-environment test-run gate below. Because the test adapter routes through the shared core, an impure dynamic field (a __init__ job) and a dynamic job generator (__dynamic__) both dispatch for a test run too, and a fixture secrets: mapping supplies additional namespaced secret contexts.

Two intentional environment gates

A test run applies two different allowLocalExecution gates, deliberately, because the two declarations mean different things:

A bound job.context is allow-and-warn. A test run never rejects on a bound environment. If a statically-named environment is a non-test environment (allowLocalExecution: false) or is not configured, it is skipped — its variables, secrets, and protection rules do not participate — and the run proceeds. A user-visible warning names the skipped environment(s), surfaced both on the kici run remote CLI output and on the dashboard run view. This keeps a job that deploys to a production environment in real runs still locally testable for its non-secret logic, while the allowLocalExecution: false boundary that keeps production secrets out of local runs is preserved (the secrets do not flow).
A fixture secrets: mapping is fail-closed. Mapping a secret context to an environment is an explicit request for that environment’s secrets. If the named environment is missing or allowLocalExecution: false, the run is rejected at trigger time with Fixture secret context '<ctx>' maps to context '<env>' which does not allow test runs.

Upload storage

Test tarballs are stored in the same S3-compatible object storage as dependency caches:

Setting	Value
Bucket	Same as `KICI_STORAGE_BUCKET`
Prefix	`test-uploads/`
Path format	`test-uploads/{routing-key}/{sha}/{upload-id}.tar.gz.enc`
Retention	24 hours (S3 lifecycle rule on prefix)
Encryption	Client-side ECDH + AES-256-GCM (described above)

No additional bucket configuration is required — operators only need to ensure the test-uploads/ prefix has a 24-hour lifecycle rule.