Test run architecture

This document describes the end-to-end data flow for remote test runs triggered by kici run remote, including the upload encryption scheme, overlay application, observer streaming, and how test runs integrate with the existing production pipeline.

High-level data flow

Developer workstation            Orchestrator              Agent
       |                              |                      |
  1. kici run remote push-main        |                      |
       |                              |                      |
  2. Compile fixture                  |                      |
       |                              |                      |
  3. POST /uploads/init  ------------>|                      |
       |<---- { signedUrl,            |                      |
       |        uploadId,             |                      |
       |        publicKey } --------- |                      |
       |                              |                      |
  4. Create overlay tarball           |                      |
       |                              |                      |
  5. Encrypt tarball (ECDH)           |                      |
       |                              |                      |
  6. PUT signed URL (S3) -----> [Object Storage]             |
       |                              |                      |
  7. POST /test/trigger  ------------>|                      |
       |<---- { runId,                |                      |
       |        observeUrl } -------- |                      |
       |                              |                      |
  8. WS /observe/:runId  ------------>|                      |
       |                              |                      |
       |                         9. Trigger match            |
       |                              |                      |
       |                        10. Dispatch -----> job.dispatch
       |                              |              (with tarballUrl,
       |                              |               cliPublicKey,
       |                              |               orchestratorPrivateKey)
       |                              |                      |
       |                              |                11. Clone repo
       |                              |                      |
       |                              |                12. Download tarball
       |                              |                      |
       |                              |                13. Decrypt (ECDH)
       |                              |                      |
       |                              |                14. Verify checksums
       |                              |                      |
       |                              |                15. Apply overlay
       |                              |                      |
       |                              |                16. Execute steps
       |                              |                      |
       |<---- observe.log ------------|<---- log.chunk ------|
       |<---- observe.step -----------|<---- step.status ----|
       |<---- observe.complete -------|<---- job.status -----|
       |                              |                      |
  17. Show summary + exit code        |                      |

Upload encryption

Test run tarballs are encrypted using ephemeral X25519 ECDH key exchange with AES-256-GCM symmetric encryption. This ensures that uploaded content is protected in transit and at rest in object storage.

Key exchange flow

    CLI                     Orchestrator                Agent
     |                           |                        |
     |  POST /uploads/init       |                        |
     |-------------------------->|                        |
     |                           |                        |
     |  Generate orchestrator    |                        |
     |  ephemeral X25519 keypair |                        |
     |                           |                        |
     |  { publicKey, uploadId }  |                        |
     |<--------------------------|                        |
     |                           |                        |
     |  Generate CLI             |                        |
     |  ephemeral X25519 keypair |                        |
     |                           |                        |
     |  ECDH shared secret:      |                        |
     |  cliPrivate + orchPublic  |                        |
     |       = AES-256 key       |                        |
     |                           |                        |
     |  Encrypt tarball          |                        |
     |  AES-256-GCM              |                        |
     |                           |                        |
     |  Upload encrypted +       |                        |
     |  send cliPublicKey        |                        |
     |-------------------------->|                        |
     |                           |                        |
     |                           |  Dispatch job with:    |
     |                           |  tarballUrl,           |
     |                           |  cliPublicKey,         |
     |                           |  orchestratorPrivateKey|
     |                           |----------------------->|
     |                           |                        |
     |                           |  ECDH shared secret:   |
     |                           |  orchPrivate + cliPub  |
     |                           |       = AES-256 key    |
     |                           |                        |
     |                           |  Decrypt tarball       |
     |                           |  AES-256-GCM           |

Crypto details

Component	Algorithm
Key exchange	X25519 (Curve25519 ECDH)
Key derivation	HKDF-SHA256 with info kici-upload-encryption
Symmetric encryption	AES-256-GCM
Wire format	[12-byte IV][16-byte auth tag][ciphertext]
Key serialization	DER format (SPKI for public, PKCS8 for private)

The shared secret is derived via crypto.diffieHellman() and stretched through HKDF to produce a 32-byte AES key. Each upload uses fresh ephemeral keypairs — keys are never reused.

Security properties

• Forward secrecy: Ephemeral keypairs mean compromising stored ciphertext later is useless without the keys (which are deleted after use)
• Integrity: AES-256-GCM auth tag prevents tampering
• No key reuse: Every upload generates fresh keypairs on both sides
• No plaintext in storage: Object storage only ever holds encrypted data

Overlay application

The agent applies the developer’s local changes on top of a fresh git clone. This produces the exact same file state as the developer’s working tree.

Application flow

1. Agent clones repo at the SHA specified in the fixture
2. Agent downloads encrypted tarball from the URL provided in the job dispatch
3. Agent derives shared secret using orchestrator’s private key + CLI’s public key
4. Agent decrypts tarball using AES-256-GCM
5. Agent extracts tar.gz to a temporary directory
6. Agent reads manifest.json from .kici-overlay-tmp/ in the extracted files
7. Agent verifies SHA256 checksums of every extracted file against the manifest
8. Agent copies files from the extracted overlay to the clone directory
9. Agent deletes files listed in the manifest’s deletions array
10. Agent cleans up temporary files

Tarball structure

overlay.tar.gz
  .kici-overlay-tmp/
    manifest.json       # Checksums, deletions, HEAD SHA
  src/
    modified-file.ts    # Changed files at their repo-relative paths
    new-file.ts
  tests/
    added-test.ts

Manifest format

{
  "sha": "abc123def456...",
  "deletions": ["src/removed-file.ts", "docs/old-guide.md"],
  "checksums": {
    "src/modified-file.ts": "sha256-hex-hash",
    "src/new-file.ts": "sha256-hex-hash",
    "tests/added-test.ts": "sha256-hex-hash"
  }
}

• sha: The HEAD commit SHA the overlay is based on. The agent clones this exact commit.
• deletions: Files the developer deleted locally. The agent removes these from the clone.
• checksums: SHA256 hashes of each included file. The agent verifies these after extraction to detect corruption.

Observer WebSocket channel

The CLI connects to a read-only observer WebSocket endpoint to receive real-time updates during execution.

Connection flow

1. CLI receives observeUrl from the test trigger response
2. CLI opens WebSocket to /api/v1/observe/:runId
3. CLI authenticates with its API key token
4. Orchestrator streams events as they occur

Observer message types

Message	Direction	Content
observe.subscribe	CLI -> Orchestrator	Run ID, auth token, optional lastSeenTimestamp
observe.status	Orchestrator -> CLI	Run/job status updates (queued, running, success, failed)
observe.log	Orchestrator -> CLI	Log lines from step execution (job-prefixed)
observe.step	Orchestrator -> CLI	Step start/complete events
observe.complete	Orchestrator -> CLI	Final run result with job summary table

Reconnection and backfill

If the CLI’s WebSocket disconnects (network blip, laptop sleep):

1. CLI reconnects to the same observer endpoint
2. CLI sends observe.subscribe with lastSeenTimestamp of the last received message
3. Orchestrator replays missed log chunks and status updates from persistent storage
4. Live streaming resumes from the current point

Multiple CLI clients can observe the same run simultaneously.

Test runs vs production runs

Test runs share most of the production pipeline but differ in key ways:

Aspect	Production run	Test run
Trigger source	GitHub webhook	POST /api/v1/test/trigger
Event normalization	Provider-specific normalizer	Synthetic event from fixture
Trigger matching	Lock file triggers	Same pipeline (or bypass with --workflow)
Dispatch core	Shared dispatchMatchedWorkflow	Same shared core (needs DAG, host fan-out, dynamic)
Repo state	Exact commit from webhook	Clone + overlay of local changes
Secret access	All contexts	Only allowLocalExecution: true contexts
Tracking	execution_runs table	Same table with is_test_run = true
Delivery ID	Provider-assigned	test: prefix + UUID
Observer streaming	Not available	WS observer channel
ctx.isTestRun	false	true

One dispatch core, two adapters

There is a single dispatch core (dispatchMatchedWorkflow) and two thin adapters that feed it: the webhook adapter (real provider event) and the test adapter (processTestTrigger, for kici run). The webhook-only preamble — delivery dedup, Platform relay, provider normalization, source registration — runs in the caller, not inside the core, so the test adapter reaches the same core without it.

The test adapter:

1. Resolves the lock file (inline for a local repo with no remote, or the same provider-driven fetch the webhook uses).
2. Selects matched workflow decisions (normal trigger matching, or a direct --workflow bypass).
3. Enforces the allowLocalExecution environment gate and stores the fixture payload.
4. Builds a dispatch context per matched workflow — a synthetic WebhookInfo, the test provenance fields, and a CLI-secret overlay that wins over orchestrator env secrets — and calls the shared core.
5. Marks the execution as isTestRun for observer broadcasting and secret gating.

Because the test adapter calls the same core, a kici run exercises the full dispatch behavior — needs-DAG scheduling, runsOnAll host fan-out, matrix and fan-out edge wiring, and deferred init/dynamic job dispatch — exactly as a webhook does. A multi-job needs workflow run via kici run honors the dependency DAG (a downstream job dispatches only after its upstream reaches a matching state), and a runsOnAll job fans out to one pinned execution per matching roster host.

Dispatch parity

A test-run job.dispatch carries the same execution-shaping fields as a production dispatch, all derived from the fixture’s simulated event: the normalized event envelope ({ type, action, targetBranch, sourceBranch, changedFiles, payload, … }), the resolved job env, the resolved environment name, and that environment’s variables. Dynamic functions evaluate against this envelope exactly as they do in production. A pure inline environment expression is evaluated at the orchestrator and its resolved name is subject to the bound-environment test-run gate below. Because the test adapter routes through the shared core, an impure dynamic field (a __init__ job) and a dynamic job generator (__dynamic__) both dispatch for a test run too, and a fixture secrets: mapping supplies additional namespaced secret contexts.

Two intentional environment gates

A test run applies two different allowLocalExecution gates, deliberately, because the two declarations mean different things:

• A bound job.context is allow-and-warn. A test run never rejects on a bound environment. If the environment (statically named or resolved from a pure inline expression) is a non-test environment (allowLocalExecution: false) or is not configured, it is skipped — its variables, secrets, and protection rules do not participate — and the run proceeds. A user-visible warning names the skipped environment(s), surfaced both on the kici run remote CLI output and on the dashboard run view. This keeps a job that deploys to a production environment in real runs still locally testable for its non-secret logic, while the allowLocalExecution: false boundary that keeps production secrets out of local runs is preserved (the secrets simply do not flow).
• A fixture secrets: mapping is fail-closed. Mapping a secret context to an environment is an explicit request for that environment’s secrets. If the named environment is missing or allowLocalExecution: false, the run is rejected at trigger time with Fixture secret context '<ctx>' maps to environment '<env>' which does not allow test runs.

Upload storage

Test tarballs are stored in the same S3-compatible object storage as dependency caches:

Setting	Value
Bucket	Same as KICI_STORAGE_BUCKET
Prefix	test-uploads/
Path format	test-uploads/{routing-key}/{sha}/{timestamp}.tar.gz
Retention	24 hours (S3 lifecycle rule on prefix)
Encryption	Client-side ECDH + AES-256-GCM (described above)

No additional bucket configuration is required — operators only need to ensure the test-uploads/ prefix has a 24-hour lifecycle rule.

See also

• Data Flows — production webhook and job execution flows
• Protocol Messages — full protocol schema reference
• Secrets Management — configuring the allowLocalExecution flag