Job execution lifecycle

When the orchestrator dispatches a job to an agent, the agent creates an ExecutionSandbox (container, bare-metal, or firecracker) and delegates execution to it. The sandbox runs customer code in an isolated child process — never in the agent’s V8 isolate. The sandbox handles cloning, dependency restore, workflow loading, rule evaluation, and step execution. IPC callbacks wire sandbox events (step status, log lines, event emissions) to the agent’s WebSocket pipeline. The entire lifecycle is managed by the JobRunner class.

See packages/agent/src/execution/job-runner.ts for the top-level orchestration and packages/agent/src/execution/sandbox/ for sandbox implementations.

Lifecycle diagram

The agent checks for cancellation (via AbortController) before sandbox creation and before execution. If job.cancel is received at any point, the sandbox is aborted and the job transitions to the cancelled state.

Five special job types skip the step sandbox, because they never execute customer workflow steps. The agent checks for them in this order:

1. Init-only jobs — dynamic field resolution.
2. Global-eval-round jobs — filters and generators for the candidate global workflows of one workflow repo, evaluated on one dual checkout.
3. DynamicJobFn evaluation jobs — runtime job generation.
4. Bring-up jobs — init-runner SSH bring-up. No clone, no child.
5. Build-only jobs — cache population.

For a global workflow, init-only and DynamicJobFn evaluation jobs use the same dual checkout as the round: the workflow repo under workflow/, where the module is loaded from, and the source repo under source/.

Four of the five still load a customer workflow module, so they run that load and the evaluation around it in the eval child (eval-runner.js), forked with a sanitized environment and a narrow IPC surface. The agent process performs the parts that are its own work: materializing the workspace, installing dependencies, packing and uploading a tarball, and reporting status. Only bring-up jobs load nothing.

Each converges on the same teardown path as a standard job.

Agent-level steps

The agent’s JobRunner.runJob() manages the outer lifecycle. Customer code runs in a child process, never in the agent: step bodies in the step runner, and every module load and evaluation in the eval child. Both children are forked with a sanitized environment that carries no agent credential, and the agent removes its own credentials from process.env after boot.

Step 1: Sandbox selection

The execution mode is determined from the job config and environment (see Container execution mode for priority rules). The agent resolves the path to the compiled workflow-runner.js entry point and builds a sanitized environment for the sandbox.

Step 2: Sandbox setup

The agent calls sandbox.setup() which prepares the execution environment:

• Container: docker create + docker start with the workflow runner bind-mounted
• Bare-metal: Validates paths and environment
• Firecracker: VM is already booted by the scaler; agent runs inside

Step 3: Job context emission

Before sandbox execution, the agent sends a job.context message to the orchestrator with runtime details (Node version, OS, arch, sandbox type, working directory, git ref, environment variables). This enables the dashboard Summary tab.

Step 4: Sandbox execution

The agent calls sandbox.executeJob() which delegates to the workflow runner child process. IPC callbacks wire sandbox events to the WebSocket pipeline:

• onStepStatus — forwarded as step.status messages
• onLogLine — batched by LogStreamer and sent as log.chunk messages
• onEventEmit — relayed to orchestrator as event.emit, response returned via IPC
• onConcurrencyReport — relayed as job.concurrency.report, ack returned via IPC

Sandbox-internal pipeline

Inside the sandbox child process (workflow runner), the following steps execute sequentially:

Repository clone

The sandbox performs a shallow git clone at the dispatch ref. For private repositories, a GitHub App installation token from job.dispatch.token is injected via http.extraHeader git config (keeps it out of the clone URL). After clone, HEAD is verified against the expected SHA.

Overlay (test runs only). For kici run remote runs, the sandbox downloads, decrypts, and applies the developer’s uncommitted changes on top of the clone. See Test run architecture.

See packages/agent/src/checkout/git-clone.ts for the implementation.

Source and dependency restore

On execution jobs the sandbox performs exactly two cache fetches:

1. Source tarball — downloads the .kici/ source tarball at sourceTarUrl, verifies its bytes against sourceTarDigest, and replaces the clone’s .kici/ with the restored tree, so a file deleted from .kici/ does not survive a cache hit and the loaded workflow’s contentHash matches the lock file exactly. The tarball replaces the install-and-compile of .kici/, not the checkout: the clone above still runs unless the job sets checkout: false.
2. Dependency tarball — downloads the node_modules tarball at depsUrl to a temporary file, verifies its SHA-256 against depsHash, and only then extracts it into .kici/node_modules/. If depsUrl is absent (dep cache miss), or the restore fails for a reason other than a hash mismatch, the sandbox installs the dependencies inline instead.

See Source and dependency caching flow for the full cache architecture.

Workflow loading

The sandbox registers a Node ESM loader hook that transforms TypeScript on import, once per process, and then dynamic-import()s the workflow .ts file directly from the extracted source tree. Which hook it registers depends on the backend: the bare-metal and firecracker sandboxes register @kici-dev/core/ts-loader-hook by bare specifier, resolved through the workspace node_modules they bind read-only; the container sandbox sets KICI_TS_LOADER_HOOK_PATH to the pure-JS hook bundle it mounts at /opt/kici/ts-loader-hook.js, and the runner registers that by absolute file:// URL, because a bare specifier would need a node_modules tree next to the runner bundle and a bare job container has none. There is no runtime bundler — TypeScript is transformed on import, and transitive imports (@kici-dev/sdk, host-repo helpers) resolve against .kici/node_modules/ via Node’s normal ESM lookup.

The lock file’s source.file and source.exportName identify the exact workflow and export to load from the imported module.

See packages/agent/src/execution/workflow-loader.ts, packages/core/src/ts-loader-hook.ts, and packages/agent/src/execution/sandbox/container-ts-loader-hook.ts for the implementation.

Process:

1. Resolve source file path relative to work directory
2. Register the TS loader hook (idempotent, once per process)
3. Walk the extracted .kici/ tree, compute contentHash = SHA-256(COMPILE_SCHEMA_VERSION + ":" + treeDigest [+ "\0" + assetDigest]) (with COMPILE_SCHEMA_VERSION = 7), and compare to the lock file’s expected contentHash. treeDigest covers every file under .kici/ except the paths .kici/.kiciignore declares, so an edit to any file the workflow imports is caught. The digest normalizes line endings (CRLF → LF) with the same implementation the compiler uses, so an LF-authored lockfile matches a CRLF working tree on Windows hosts (where Git’s system gitconfig defaults core.autocrlf=true). A mismatch throws a “lock file is out of date” error carrying the baked agent SDK version + bundle hash for debugging.
4. Dynamic-import() the .ts file (cache-busted with a ?t=<ts> query) — the loader hook transforms TS on demand.
5. Extract the named workflow from module exports.

Workflow and step extraction

After loading the module, the sandbox extracts the target workflow by export name and resolves its steps, normalizing bare functions and step() calls into a uniform structure. Step names are assigned at compile time during lock file generation — unnamed steps receive counter IDs (step-1, step-2, etc.) in the compiler, not at runtime. Concurrency group evaluation also happens at this stage.

Rule evaluation

Rules are evaluated sequentially with fail-fast behavior. Each rule’s check function is called with a RuleContext providing the event payload, changed files, environment variables, and a zx $ shell. Rules can be synchronous or asynchronous.

If a rule check returns false, the job is skipped (not failed): it reports skipped with every step skipped, and its error names the rule (rule '<label>' returned false). No step starts, so no step.status is sent; the orchestrator records the step rows from the stepResults of that terminal job.status, as it does for any job that streamed no step. To an orchestrator that does not advertise acceptsSkippedJobStatus on register.ack, the agent reports success instead. If a rule check throws, the job fails with the error, because a rule that could not decide must not read as “do not run”. In both cases the remaining rules are not evaluated. Rule evaluation results (label, passed, duration, error) are included in the job.status message data.

See packages/agent/src/execution/rule-evaluator.ts for the implementation.

Hook collection

The sandbox collects job-level hooks (beforeStep, afterStep, onSuccess, onFailure, onCancel, cleanup) from the loaded workflow module. These hooks are invoked at the appropriate points during step execution — beforeStep/afterStep around each step, onSuccess/onFailure after all steps complete, onCancel on cancellation, and cleanup always runs last.

Per-job init phase

When a job declares init (see Per-job init), an init phase runs after the hooks are collected and before the step loop begins — provisioning a repo-declared toolchain so it is on the environment every step sees. Each init spec runs through the same sandbox shell that steps use (cwd at the clone root, the job’s environment, masked log streaming), and each surfaces as an init:<n> pseudo-step in the run timeline alongside the step list (mirroring how hook:* rows appear).

Typed presets ('mise' / { mise } / { apt }) and 'auto' detection are expanded into concrete init specs at the agent init phase, where the clone root is on disk to hash for the cache key and scan for marker files. A preset registry maps each preset name to an OS-branched expander, and an ordered auto-detect table maps marker files to presets. Both produce the same generic init spec that the phase below loops over, so the run/cache/env/timeout machinery is unchanged.

For each init spec, in declaration order:

1. Cache restore — if the spec sets cache, the agent restores the cached paths before the command, so a warm key skips the download.
2. Run the command — the init command runs in the sandbox shell against a per-init wall-clock timeout. Before it runs, the agent allocates fresh $KICI_ENV / $KICI_PATH files and points the shell’s environment at them. The command’s environment is the job’s environment with the spec’s env laid over it, for that command only. Keys that start with KICI_ and operator secret keys keep the job’s value, and the init step’s log names each refused key.
3. Apply the env / PATH delta — after the command succeeds, the agent reads the KEY=value lines from $KICI_ENV and the directory lines from $KICI_PATH, applies the delta (PATH entries prepended), and the result becomes visible to every subsequent init and step.
4. Cache save — if the spec set cache and the key missed on restore, the agent saves the cached paths after the command.

If any init exits non-zero or exceeds its timeout, the job fails before any step runs: the init:<n> pseudo-step is marked failed with its logs attached, the remaining inits and the entire step loop are skipped, and the job reports a terminal failure (a timeout is reported with the distinct timeout reason). This makes a broken toolchain an early, attributable failure rather than a confusing mid-step error.

Step execution

Steps execute sequentially within a job. Each step receives a full StepContext with:

• $ — zx shell (runs natively inside the sandbox process, regardless of sandbox type)
• log — step-scoped logger that streams via IPC to the agent, then to the orchestrator
• env — merged environment variables (process env + job env + step env + secrets)
• ctx — workflow and job metadata (names, runsOn label)
• matrix — optional matrix values for the current combination

Each step runs inside a Promise.race against a timeout. The timeout is configured per-step (step.timeout) or falls back to the agent’s default (KICI_DEFAULT_STEP_TIMEOUT_MS, default 30 minutes).

A run carries a check mode (apply | check | check-fail-on-drift) that threads from the CLI through dispatch to the step loop. For a step that declares a check facet, the loop reuses the shared check / apply primitive: in apply mode it converges (drift ⇒ apply ⇒ applied; null ⇒ in-sync skip), and in check mode it previews drift without applying (would change). A plain step under check mode is skipped (no_check). Under check-fail-on-drift, any previewed drift makes the run terminal status failed. See Idempotent steps and check mode.

See packages/agent/src/execution/sandbox/workflow-runner.ts for the implementation.

Step lifecycle:

1. Emit step.status: running via IPC (agent forwards to orchestrator)
2. Execute step.run(ctx) with timeout enforcement
3. Emit step.status: success or step.status: failed with duration and diagnostics
4. Return StepResult with continueOnError flag

Step failure behavior:

• If continueOnError is false (default): job execution stops immediately
• If continueOnError is true: execution continues to the next step, but the final job status still reflects the failure

Timeout behavior:

• Step cancelled via AbortController after the configured timeout
• Error message includes step name and timeout duration

Log streaming

Logs flow from the sandbox child process via IPC to the agent, where the LogStreamer batches output lines and sends them to the orchestrator as log.chunk WebSocket messages. Each step has its own LogStreamer instance, created lazily on first log line.

See packages/agent/src/execution/log-streamer.ts for the implementation.

What gets captured inside a sandbox step: the sandbox runs a three-layer capture so that everything a step writes ends up in the step log stream:

1. Subprocess output — the step-local zx $ is configured with a log callback (createSandboxStepContext in workflow-runner.ts) that intercepts every { kind: 'stdout' | 'stderr' } entry zx emits for child processes and forwards each completed line as log.line. zx pipes child stdio to an internal VoidStream, so without this override subprocess output would be invisible.
2. Direct console output from step TS/JS code — installOutputCapture() monkey-patches process.stdout.write and process.stderr.write while a step is active (captureStepIndex >= 0). Calls to console.log, console.error, console.warn, or any library that writes directly to process.stdout/stderr are split into lines and forwarded as log.line. The patch is also active during the pre-step prepare phase (module load, concurrency-group evaluation, rule evaluation) under capturePrepareActive = true, routing output to the workflow-level log bucket (stepIndex: -1).
3. Structured logger — the step context’s log object (log.info / log.warn / log.error / log.debug) sends log.line directly over IPC, bypassing the stdio hooks.

Every captured line carries the stream it came from (stdout or stderr) from the zx callback through the IPC message and the chunk batcher, out on the log.chunk message, and into the persisted log record’s level. The batcher closes the pending chunk whenever the stream flips, so stdout/stderr interleaving survives batching. A peer that does not send the field is read as stdout.

A failing step’s error lands in its own log. When a step body throws, the error message is emitted into that step’s log stream — tagged stderr and ordered ahead of the terminal status — instead of only riding step.complete. This matters because subprocess wrappers pack the real diagnosis (captured stderr, exit code, kill signal) into the thrown error’s message. The logged copy is bounded to 100 lines / 8192 characters and states what it dropped; the full untruncated message still travels on step.complete. It goes through the same send path as every other log line, so secret masking applies unchanged.

All three paths converge in the agent’s per-step LogStreamer. Hooks (beforeStep, afterStep, onSuccess, onFailure, onCancel, cleanup) also run under captureStepIndex — per-step hooks reuse the step index and land in the step’s log; post-loop and cancel-path hooks allocate their own indices above steps.length and appear as dedicated step rows in the dashboard.

What gets captured inside an evaluation job (__init__ / __build__ / __dynamic__, and a global eval round): these jobs don’t use the step sandbox. Their module load and evaluation run in the eval child. A second capture primitive — AsyncLocalStorage-scoped console.* patching in packages/agent/src/execution/console-capture.ts — routes user-supplied output over IPC to a per-job synthetic step-0 LogStreamer in the agent.

The captured surfaces are:

• console.log / console.error / console.warn / console.info / console.debug from module top-level code.

• Dynamic context / env / concurrencyGroup functions.

• The DynamicJobFn body and generated-job env / matrix functions.

• Subprocess output from the per-invocation zx $ callback installed on the DynamicJobFn path:

  await $`echo hello`; // captured via the zx log callback

The patch targets only console.* methods (not process.stdout.write) because the agent’s structured logger writes directly to stdout. Capturing stdout at the agent level would leak agent-internal log output into user step streams.

What is still not captured:

• Direct process.stdout.write or printf inside evaluation jobs — use console.* or the provided log parameter instead.
• Output outside a capture scope — in the sandbox this means any IPC or setup code that runs before installOutputCapture() returns; in the eval child this means anything on an async stack not descended from a runCaptured(...) call.

Flush triggers:

• Line count reaches threshold (default: 50 lines)
• Timer fires (default: 100ms)
• Step completes (explicit flush)

Size limit:

• Maximum log size per step is controlled by maxLogSizeBytes from the job dispatch message, falling back to KICI_MAX_LOG_SIZE_BYTES (default: 10 MB)
• After exceeding the limit, a [TRUNCATED: log output exceeded <maxLogSizeBytes> bytes] notice is sent and further lines are silently dropped

Message format:

log.chunk {
  messageId, runId, jobId, stepIndex,
  lines: string[],
  timestamp,
  stream: 'stdout' | 'stderr'
}

Log chunks are sent via the buffered send() path (not sendDirect()), so they participate in event buffering during disconnection.

Status reporting

The agent reports status at each lifecycle point using sendDirect(), which bypasses the event buffer to ensure protocol messages are delivered immediately when the connection is available. Log chunks use the buffered send() path to participate in event buffering during disconnection.

Message	When	Key data
job.status: running	Job starts	runId, jobId
job.context	Before sandbox execution	runtime, sandboxType, envVars
job.heartbeat	Periodic during execution	runId, jobId, timestamp
step.status: running	Each step starts (via IPC)	stepIndex, stepName
step.status: success/failed/skipped	Each step completes (via IPC); skipped when its rule returns false	durationMs, exitCode, signal
log.chunk	During execution (via IPC)	lines (batched)
job.status: success	All steps pass	durationMs, stepResults
job.status: failed	Any step fails	durationMs, stepResults, error
job.status: cancelled	Abort signal	—
job.status: skipped	Rule check returns false	durationMs, stepResults (all skipped), error

Cleanup

After execution completes (regardless of outcome), the sandbox is torn down via sandbox.teardown(), the work directory is removed via fs.rm() with recursive: true, force: true, and the active job slot is freed from the JobRunner.activeJobs map.

Cleanup runs in a .finally() block to ensure it executes even if the job throws an unexpected error.

Container execution mode

The execution mode is determined at the job level (not per-step) by the JobRunner using the following priority:

1. KICI_JOB_IMAGE_AGENT=1 → bare-metal (the agent already runs inside the job’s own image, so it runs the steps directly)
2. Container config in job dispatch → container
3. KICI_EXECUTION_MODE env var → explicit override
4. KICI_SCALER_MANAGED=1 → firecracker
5. Default → bare-metal

When running in container mode, the agent uses the ContainerSandbox. The workflow runner is bind-mounted read-only into the container as a self-contained bundle (its runtime dependencies inlined, plus a pure-JS TypeScript loader-hook bundle mounted alongside it) so the single-file runner loads inside a bare job container with no access to the agent’s node_modules. IPC uses stdin/stdout JSON-lines over the container runtime’s exec API.

See packages/agent/src/execution/sandbox/container-sandbox.ts for the implementation.

Container lifecycle:

1. Prepare the workspace on the host — the agent clones the repository (or, for a full-repo test run, applies the overlay) and builds the job image when the job declares a Dockerfile. It then installs .kici/ dependencies into the checkout when lifecycle scripts are disabled, a runtime is injected and the dispatch carries no dependency cache. .kici/ must also pass the host-install allowlist (packages/agent/src/execution/host-deps-install.ts, host-install-eligibility.ts), and an npm older than 11.15.0, which runs npm ci, needs a lockfile that pins every package to a semver version and a registry tarball (host-install-lockfile.ts). The tree is copied into /workspace after the container starts. The clone and the install write their progress to the job’s workflow-level log (step -1), the same log the runner writes its own setup lines to; the image build reports as its own container:build step
2. Create container — docker create with the self-contained workflow-runner bundle bind-mounted read-only at /opt/kici/workflow-runner.js (and the loader-hook bundle at /opt/kici/ts-loader-hook.js), pre-sanitized environment variables, and sleep infinity as the entrypoint to keep the container alive. /workspace is a container-owned anonymous volume created fresh per job and owned by the container user — not a host bind — so clone/install/execute writes work even with all capabilities dropped under rootful container runtimes
3. Start container — docker start
4. Execute job — docker exec runs the workflow runner inside the container; the runner handles workflow loading and step execution, plus the dependency install when the host did not do it
5. Cleanup — docker rm -f after job completes

Important: With the sandbox model, the entire workflow runner process (including step TypeScript code and shell commands via $) runs inside the container. Agent-internal credentials (KICI_*, KICI_DATABASE_URL, etc.) never enter the container — only pre-sanitized environment variables are passed through.

Debug mode: When KICI_DOCKER_KEEP_FAILED=true, failed job containers are preserved for debugging instead of being removed.

Concurrency control

The agent enforces single-job concurrency: only one job runs at a time. When a job.dispatch arrives and the agent is already running a job, the agent sends job.reject with reason busy, then an agent.status with its current counts. The explicit reject lets the orchestrator requeue the job at once instead of leaving it bound to this agent until disconnect.

See packages/agent/src/server.ts for the concurrency gating logic.

Capacity flow:

1. job.dispatch arrives
2. If the agent is draining: send job.reject with reason draining, then agent.status
3. If jobRunner.activeJobs.size > 0: send job.reject with reason busy, then agent.status
4. If idle: send job.ack (before any clone work, so a slow clone cannot use up the orchestrator’s acknowledgement deadline), then execute
5. On job completion: send agent.status with updated counts

The container scaler sets KICI_SCALER_SINGLE_JOB=1 on an agent container it starts from the label set’s agent image rather than a per-job image. Such a single-job agent enters drain mode the moment it accepts its job and reports the drain on agent.status, so the orchestrator never offers it a second job. It exits when the job ends.

Cancellation

When the orchestrator sends job.cancel, the agent:

1. Looks up the job in activeJobs by jobId
2. Calls abortController.abort(reason) on the active job
3. The running phase detects the abort signal at the next check point
4. If a step is mid-execution, the step’s process receives SIGTERM
5. After the grace period (default 30s for bare-metal/firecracker, 10s for container), the agent force-kills with SIGKILL
6. Reports job.status: cancelled
7. Cleans up work directory

See packages/agent/src/server.ts for the SIGTERM/SIGUSR1 shutdown handlers.

Drain mode (SIGUSR1):

The agent supports a drain mode for graceful scaling down. When SIGUSR1 is received:

1. Stop accepting new job dispatches, and report the drain on agent.status so the orchestrator stops offering jobs
2. Wait for all active jobs to complete (polls every 1 second)
3. Stop metrics reporter (final metrics flush)
4. Disconnect from orchestrator
5. Stop HTTP server
6. On a scaler-managed agent, wait 1 second so the WebSocket close frame flushes
7. Exit

See also

• Reconnection and event buffering — WebSocket reconnection behavior during agent disconnection
• Protocol messages — message schemas for job.dispatch, job.status, log.chunk
• Execution status vocabulary — run, job, and step status vocabularies and terminal states
• Agent configuration — environment variables for the agent
• Agent getting started — deployment guide