# KiCI Workflow features: execution

This bundle covers: Concurrency, dynamic values, events, container jobs, environment variables, global workflows, idempotent steps.

## Concurrency groups

Source: https://docs.kici.dev/user/concurrency/

Concurrency groups prevent multiple workflow runs from executing in parallel when they target the same resource. Common use cases include preventing parallel deploys to the same environment or serializing database migrations.

## Basic usage

```typescript
import { workflow, job, step, push } from '@kici-dev/sdk';

export default workflow('deploy', {
  on: push({ branches: ['main', 'staging'] }),
  concurrency: {
    group: (ctx) => `deploy-${ctx.branch}`,
    cancelInProgress: true,
    max: 1,
  },
  jobs: [
    job('deploy', {
      runsOn: 'linux',
      steps: [
        step('deploy', async ({ $ }) => {
          await $`./deploy.sh`;
        }),
      ],
    }),
  ],
});
```

## Configuration

The `concurrency` option on a workflow accepts:

| Field              | Type     | Default  | Description                                |
| ------------------ | -------- | -------- | ------------------------------------------ |
| `group`            | Function | Required | Returns the concurrency group key string   |
| `cancelInProgress` | boolean  | `true`   | Cancel older runs when a newer run arrives |
| `max`              | number   | `1`      | Maximum concurrent runs in the same group  |

### Group key function

The group key function receives a context with the branch name and event payload. Runs with the same group key are subject to concurrency limits.

```typescript
// Per-branch concurrency (most common)
group: (ctx) => `deploy-${ctx.branch}`;

// Global concurrency (across all branches)
group: () => 'deploy';

// Per-target-branch concurrency
group: (ctx) => `deploy-${ctx.event.targetBranch ?? 'default'}`;
```

The workflow-level group function is always evaluated **agent-side** at runtime -- the lock file records only that a group function exists (`hasGroup: true`), not the function itself. The agent loads the workflow source, calls the group function with `{ branch, event }`, and reports the evaluated key back to the orchestrator before step execution begins. Job-level `concurrencyGroup` functions (see [Contexts](https://docs.kici.dev/user/contexts/#concurrency-groups)) resolve the same way — on the agent's init step, never in the orchestrator.

## cancelInProgress mode

When `cancelInProgress: true`, a newer run supersedes older runs in the same group:

```
Run #1 starts deploying to main        -> running
Run #2 arrives for deploy-main group   -> Run #1 cancelled ("Superseded by run in concurrency group 'deploy-main'")
Run #2 continues                       -> running
```

This is the most common mode for deploy workflows -- you want the latest code deployed, not an outdated version.

The cancelled run:

- Receives a cancellation with reason "Superseded by run in concurrency group 'deploy-main'"
- Goes through the normal cancel flow (grace period, hooks if graceful)
- GitHub Check status updated to `cancelled` with the superseded reason

```typescript
workflow('deploy', {
  concurrency: {
    group: (ctx) => `deploy-${ctx.branch}`,
    cancelInProgress: true,
  },
  jobs: [/* ... */],
});
```

## Queue mode

When `cancelInProgress: false`, newer runs will wait until older runs complete:

```
Run #1 starts deploying                -> running
Run #2 arrives for same group          -> queued ("Waiting for deploy-main (1 ahead)")
Run #1 completes                       -> success
Run #2 starts                          -> running
```

In queue mode, the agent that picked up the queued run **stays connected** to the orchestrator and parks on a long-poll wait. When the holder finishes (success, failure, or cancel), the orchestrator dequeues the FIFO-next entry and pushes a `proceed` notification over the same WebSocket; the queued agent then continues with normal step execution against the workspace it already has. The agent's slot is therefore held for the duration of the queue wait — bound by `KICI_CONCURRENCY_WAIT_TIMEOUT_MS` (default 1 hour).

```typescript
workflow('migrate-db', {
  concurrency: {
    group: () => 'migrations',
    cancelInProgress: false,
    max: 1,
  },
  jobs: [/* ... */],
});
```

The dashboard will show a "Queued" badge with the reason: "Waiting for deploy-main (1 ahead)".

## Max concurrent runs

The `max` field controls how many runs can execute simultaneously in the same group:

```typescript
// Allow up to 3 parallel test runs per branch
workflow('test', {
  concurrency: {
    group: (ctx) => `test-${ctx.branch}`,
    cancelInProgress: false,
    max: 3,
  },
  jobs: [/* ... */],
});
```

When `max: 1` (default), runs are fully serialized within the group.

`max` is enforced by the orchestrator's database, so the limit is cluster-wide and survives a restart. A run holding a slot keeps it across an orchestrator restart. Every orchestrator in a cluster counts against the same limit, so a group with `max: 1` runs one job at a time no matter which orchestrator dispatched it.

## Group key examples

### Deploy per environment

```typescript
workflow('deploy', {
  concurrency: {
    group: (ctx) => `deploy-${ctx.branch}`,
    cancelInProgress: true,
  },
  jobs: [
    job('deploy-staging', {
      runsOn: 'linux',
      context: 'staging',
      steps: [/* ... */],
    }),
  ],
});
```

### Global singleton

```typescript
// Only one migration can run at a time, regardless of branch
workflow('migrate', {
  concurrency: {
    group: () => 'db-migration',
    cancelInProgress: false,
  },
  jobs: [/* ... */],
});
```

### Environment-aware groups

```typescript
// Serialize deploys per environment
workflow('deploy', {
  concurrency: {
    group: (ctx) => {
      const env = ctx.branch === 'main' ? 'production' : 'staging';
      return `deploy-${env}`;
    },
    cancelInProgress: true,
  },
  jobs: [/* ... */],
});
```

## Interaction with context protection

When a workflow has both `concurrency` and `context` protection rules:

1. Context protection gates (required reviewers, wait timer) apply first
2. Concurrency group check happens after protection gates pass
3. If the run is queued by concurrency, it keeps its protection approval

This means a run that passed approval won't need re-approval if it gets queued by concurrency.

The two caps also differ in strength. A workflow-level concurrency group claims
its slot inside a single database transaction, so two runs that arrive together
cannot both take it. A context
[concurrency limit](https://docs.kici.dev/user/contexts/#concurrency-limits) is a throughput control:
simultaneous arrivals can each be admitted before either is recorded. Declare a
concurrency group for anything that must never run beside itself, whatever the
context limit says.

## Cancelling queued runs

Queued runs can be cancelled before they start executing. The cancel request removes them from the queue immediately -- they don't go through the grace period since no step is running.

## Job-level concurrency groups

In addition to workflow-level concurrency, individual jobs can define their own concurrency group via the `concurrencyGroup` property. This controls concurrent execution at the job level rather than the workflow level. See [Contexts — concurrency groups](https://docs.kici.dev/user/contexts/#concurrency-groups) for details.

## Local execution

`kici run <event> --local` is a real routed dispatch: your machine becomes an ephemeral agent behind the local dev plane, whose own orchestrator applies the same concurrency machinery described above. The `group` callback is evaluated agent-side against the simulated event, and `cancelInProgress` carries its usual semantics — `true` supersedes the older run in the group, `false` queues the newer one behind it.

Coordination is scoped to that plane. The plane's state (including its database) lives under `~/.kici/local/`, so enforcement is per-machine and per-user: running the same workflow on two different machines does not serialize across them. For cross-host enforcement (queueing across agents, dashboard visibility), use `kici run remote` against a deployed orchestrator.

See [`kici run <event> --local`](https://docs.kici.dev/user/cli/runs-and-approvals/#kici-run-event---local) for the rest of the local-run behavior, and [the local dev plane](https://docs.kici.dev/operator/orchestrator/local-dev-plane/) for the plane's state directory and lifecycle.

---

_Source: `packages/sdk/src/types.ts` (WorkflowOptions.concurrency, JobOptions.concurrencyGroup)_

---

## Container jobs

Source: https://docs.kici.dev/user/container-jobs/

A job can run inside a container image you choose. Set `container` on the job:

```typescript
job('build', {
  runsOn: ['kici:os:linux'],
  container: 'python:3.12-slim',
  steps: [compile, test],
});
```

Every step then runs inside that image.

## Your image needs almost nothing

KiCI supplies its own runtime. It mounts a Node build and the step runner into
the container, read-only, and runs the steps with that Node. In most setups it
also clones your repository outside the image and copies the tree in.

So the image does **not** need Node, and does **not** need npm.

### What the image must provide

Every container job needs these two:

| Requirement                                                                                                                | Why                                                   | When it is checked    |
| -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | --------------------- |
| **A GNU C library (glibc)** — the loader at `/lib64/ld-linux-x86-64.so.2` on x64, or `/lib/ld-linux-aarch64.so.1` on arm64 | The Node build KiCI mounts in is linked against glibc | Before the job starts |
| **A shell at `/bin/sh`**                                                                                                   | Steps run shell commands                              | Before the job starts |

If the image fails either, the job fails immediately and the error names the
reason — you do not wait for a build to reach its first step.

Two more apply only when your pool runs your image **as the agent**. See
[Which pool runs your image how](https://docs.kici.dev/user/container-jobs/#which-pool-runs-your-image-how) to find out
which case you are in:

| Requirement          | Why                                                                       | When it is checked |
| -------------------- | ------------------------------------------------------------------------- | ------------------ |
| **`git` on `PATH`**  | The agent clones your repository, and here the agent is inside your image | At agent startup   |
| **`bash` on `PATH`** | The agent needs it to run steps                                           | At agent startup   |

These two are not checked ahead of time. The agent fails to start, and the job
waits for an agent that never arrives.

Most build images provide all four. `python`, `golang`, `node` and `rust` ship
git and bash; their `-slim` variants often drop git — `python:3.12-slim` is one
that does.

### Alpine and other musl images

Alpine uses musl instead of glibc, so KiCI refuses it:

```
image 'alpine:3.20' uses musl libc; the musl runtime variant is not enabled
(glibc images only in this version). Use a glibc image — for example the
'-slim' rather than the '-alpine' tag.
```

Pick the glibc build of the same image. `python:3.12-slim` instead of
`python:3.12-alpine`, `node:24-slim` instead of `node:24-alpine`.

### Which pool runs your image how

A pool runs your image one of two ways, and they ask for different things.

- **The agent stays outside your image** and starts a second container from it.
  The agent clones and copies the tree in, so your image needs only glibc and a
  shell. A bare-metal pool works this way, and so does any agent that has its
  own container runtime.
- **The agent runs inside your image.** A container pool does this for a job
  that names its own image: your image becomes the agent. Your image then needs
  git and bash as well.

The second form fails at agent startup with:

```
Agent required-tools validation failed:
  - 'git' not found on PATH — required for repository checkout
This agent runs inside the job's own container image, so that image must provide
these tools. Either add them to the image, or run the job on a pool whose agent
stays outside it.
```

If your image lacks git or bash, either add them, or run the job on a pool of
the first kind.

## Build the image from a Dockerfile

Point `container` at a Dockerfile in your repository instead of naming an image.
KiCI builds it before the job starts, and runs the job in the result:

```typescript
job('build', {
  runsOn: ['kici:os:linux'],
  container: {
    dockerfile: '.kici/ci.Dockerfile',
    context: '.', // default: the repository root
    target: 'ci', // optional build stage
    args: { NODE_VERSION: '24' },
  },
  steps: [compile],
});
```

Set `image` or `dockerfile`, never both. KiCI rejects a workflow that sets both,
or neither, when you define it.

The build runs on the agent that runs the job, after it clones your repository.
So the build context is the tree at the commit that triggered the run, and your
Dockerfile's `COPY` sees exactly that code. `.dockerignore` applies as usual.

### What is reused between runs

The image is built every run. Your container runtime's layer cache does the
work, exactly as it does on your own machine: a run that changes nothing below a
`COPY` reuses those layers and finishes in seconds.

Nothing is uploaded. The image lives on the agent host that built it, and KiCI
removes the tag after the job. A different host, or a host whose cache was
pruned, builds again.

### Build arguments are not secrets

A build argument is recorded in the built image's history, so anyone who can
read the image can read the value. `args` therefore takes plain strings only —
you cannot point one at a secret. Pass a secret to a step instead.

### Who may build

A Dockerfile build runs your `RUN` commands on the agent host, **outside** the
sandbox that contains your job's steps. So KiCI refuses one on an untrusted
ref — a fork pull request, or a contributor whose access it cannot confirm.

Your operator allows it per organization:

```bash
kici-admin org-settings allow-untrusted-dockerfile-builds true --org <org>
```

A trusted ref — a push to your default branch, or a pull request from a
contributor with write access — builds without that setting.

A schedule fire and the [auto-scaler's](https://docs.kici.dev/user/workflows/autoscaling-workflows/)
`kici.scaler.scale-up` / `kici.scaler.scale-down` events are trusted refs. No run
causes them. So a cron-fired Dockerfile job builds, and so does one in a
provisioning or teardown workflow.

Every other internal trigger runs at the
[trust tier](https://docs.kici.dev/user/events/#trust-tiers-on-internal-triggers) of the run that caused
it. A workflow triggered by `ctx.emit()` carries the emitting run's tier. A
workflow triggered by a run completing carries that run's tier. So a completion
or an emit from an untrusted ref cannot build a Dockerfile.

### Requirements

The agent host needs `docker` or `podman` on its `PATH`, not only a container
runtime socket. Agents report this themselves, and KiCI routes a Dockerfile job
only to one that can build — so a pool without a CLI is skipped rather than
failing your job partway. The built image itself needs only what
[any container job's image needs](https://docs.kici.dev/user/container-jobs/#what-the-image-must-provide): a glibc and
`/bin/sh`. It never needs git or bash, because a built image is always run by an
agent that stays outside it.

## Private images

Point `auth` at secrets that hold the registry credentials. Every value is the
**name** of a secret, in `<context>:<secret-name>` form — the same form
`gitCredentials` uses:

```typescript
job('build', {
  runsOn: ['kici:os:linux'],
  container: {
    image: 'reg.internal:5000/acme/ci:1.2',
    auth: { usernameSecret: 'prod:REGISTRY_USER', tokenSecret: 'prod:REGISTRY_TOKEN' },
  },
  steps: [compile],
});
```

Store the secrets first with `kici-admin secret set`. Pasting a token straight
into the workflow is rejected when the workflow is defined, because a token
written into `.kici/` would be committed to your repository.

The named context's protection rules run before the secret is read, exactly as
they do for [git credentials](https://docs.kici.dev/user/patterns/git-credentials/#what-a-job-may-ask-for).
A `prod:` reference from a branch the `prod` context restricts is refused, and
the job is dispatched with no registry credentials — so a private image fails to
pull rather than being pulled from a branch the context does not allow. The rule
that refused it is named in your orchestrator's log, not in the run.

**An untrusted ref receives no registry credentials.** A fork pull request is
dispatched without them, so a private base image fails to pull and a public one
is unaffected. The run's reduced-privilege note says so.

The username is not a secret, so you may write it directly:

```typescript
auth: { username: 'ci-bot', tokenSecret: 'prod:REGISTRY_TOKEN' }
```

Your orchestrator resolves these names at dispatch and sends only the resolved
credentials to the agent. The agent never reads your secret store.

### Naming the registry

`auth` also takes a `registry` — the registry host the credentials belong to,
such as `reg.internal:5000`.

With `image` it is optional, because KiCI reads the host off the image
reference. With `dockerfile` it is **required**: the base image is named inside
your Dockerfile, so there is nothing to read it from. A `dockerfile` job whose
`auth` omits `registry` is refused when you define the workflow.

```typescript
container: {
  dockerfile: '.kici/ci.Dockerfile',
  auth: {
    registry: 'reg.internal:5000',
    usernameSecret: 'prod:REGISTRY_USER',
    tokenSecret: 'prod:REGISTRY_TOKEN',
  },
},
```

With `dockerfile`, these credentials pull the Dockerfile's own `FROM` base — not
a job image, since the job image is the one KiCI builds.

### Credentials that only exist at run time

A token fetched during the run — from a cloud registry's login command, for
example — has no secret to name. Use the `*Value` half of the pair instead:

```typescript
auth: { username: 'AWS', tokenValue: fetchedAtRuntime }
```

## Where container jobs run

A container job needs a container runtime on the host that runs it. KiCI does
not check that for you: your orchestrator cannot see what a given agent host
has installed.

The host also needs a copy of the KiCI runtime to mount in. Every pool your
auto-scaler provisions gets one automatically, from the agent image the pool is
configured with. An agent you start by hand needs `KICI_RUNTIME_IMAGE` set to a
`kici-agent` image — see
[Agent configuration](https://docs.kici.dev/operator/agent/configuration/). Without it, the job
runs on the image's own `node`, so the image must ship one.

If some of your pools have a runtime and some do not, label them and say so on
the job:

```typescript
job('build', {
  runsOn: ['kici:os:linux', 'kici:runtime:docker'],
  container: 'python:3.12-slim',
  steps: [compile],
});
```

A job that reaches a host with no runtime fails with an error naming what is
missing, rather than running incorrectly.

## Limits worth knowing

- **glibc only.** A musl image fails the preflight. Support for musl is a
  planned follow-up.
- **The image is pulled fresh when it is not already on the host.** A large
  image costs that pull on the first job that uses it.
- **`git` inside your steps still needs git in the image.** KiCI clones for you,
  but a step that runs `git` itself uses the image's own copy.

---

## Dynamic values

Source: https://docs.kici.dev/user/dynamic-values/

Dynamic values let you compute `context`, `env`, and `concurrencyGroup` at runtime based on the incoming event. Instead of hardcoding static strings, you pass a function that receives the normalized event envelope and returns the resolved value.

```typescript
job('deploy', {
  runsOn: ['default'],
  context: (event) => event.targetBranch,
  env: (event) => ({ BRANCH: event.targetBranch }),
  concurrencyGroup: (event) => `deploy-${event.targetBranch}`,
  steps: [/* ... */],
});
```

```typescript
job('deploy', {
  runsOn: 'default',
  // One shape everywhere: branch on the normalized event type.
  context: (event) => (event.type === 'pull_request' ? 'preview' : 'production'),
  steps: [/* ... */],
});
```

## How it works

When you define a dynamic value as a function, it is resolved on the eval agent as a short **init** step that runs before the job:

1. The orchestrator dispatches a lightweight `__init__` job to an agent.
2. The agent loads the compiled workflow bundle and calls your function with the normalized event.
3. The agent reports the resolved values back to the orchestrator, which dispatches the real execution job with them applied.

This resolution appears in the run timeline as an `Init:` entry. The orchestrator never evaluates workflow code — every dynamic `context`, `env`, and `concurrencyGroup` function runs agent-side, whatever it references.

`kici preview` lists the injected `__init__` job under each affected job, so you can spot it before the first run.

**Examples:**

```typescript
// Simple branch extraction
context: (event) => event.targetBranch;

// Object literal with string operations
env: (event) => ({ BRANCH: event.targetBranch });

// Concatenation with event data
concurrencyGroup: (event) => `deploy-${event.targetBranch}`;

// Local variables and safe globals
context: (event) => {
  const parts = event.targetBranch.split('/');
  return parts[parts.length - 1];
};

// Async lookups, module access, and process/global reads all work
context: async (event) => await lookupEnv(event.targetBranch);
env: (event) => ({ DEFAULT_ENV: process.env.DEFAULT_ENV ?? 'staging' });
```

## Performance

| Value                    | Overhead  | Example                                  |
| ------------------------ | --------- | ---------------------------------------- |
| Static value             | None      | `context: 'staging'`                     |
| Dynamic value (function) | Init step | `context: (event) => event.targetBranch` |

A static value is baked into the lock file and needs no init step. A dynamic value always resolves through the agent's init step, so reach for a function only when the value genuinely depends on the event.

## Tips

- **Prefer static values when you can.** Most context and env values are the same on every event; only make them dynamic when they truly depend on the event payload.
- **Run `kici preview`** to see the injected `__init__` job listed under each affected job before your first run.
- **A runtime error in a dynamic function fails the job.** If your function throws when the init step runs it (e.g., accessing a property on `undefined`), the job fails immediately.
- **See [how your workflow code executes](https://docs.kici.dev/user/execution-model/)** for the full picture of where dynamic values run relative to rules, hooks, and step bodies.
- **The event parameter is the normalized event envelope** — the same shape rules receive as `ctx.event`: `{ type, action, targetBranch, sourceBranch, changedFiles, payload, … }` (see the [event payload reference](https://docs.kici.dev/user/sdk/event-payloads/) for the complete schema). Narrow on `event.type` (`'push'`, `'pull_request'`, `'tag'`, …) to branch per trigger kind. The raw provider webhook body is nested at `event.payload` (for GitHub pushes: `payload.ref`, `payload.after`, `payload.repository`, …).

---

## Environment variables

Source: https://docs.kici.dev/user/env-vars/

The KiCI CLI reads the following environment variables to customize its behavior. OAuth login (`kici login` without `--token`) defaults `KICI_PLATFORM_URL`, `KICI_OIDC_ISSUER`, and `KICI_OIDC_CLIENT_ID` to the hosted KiCI Platform, so `kici login` works with no configuration. Set them only to target another KiCI environment (e.g. a testing instance) or a custom OIDC provider.

## Authentication

| Variable              | Description                            | Default                                      |
| --------------------- | -------------------------------------- | -------------------------------------------- |
| `KICI_OIDC_ISSUER`    | OIDC issuer URL for authentication     | `https://auth.kici.dev/realms/kici-internal` |
| `KICI_OIDC_CLIENT_ID` | OIDC client ID for the CLI application | `kici-cli`                                   |
| `KICI_PLATFORM_URL`   | Platform API base URL                  | `https://api.kici.dev`                       |
| `KICI_CONFIG_DIR`     | Override the KiCI config directory     | `~/.kici`                                    |

## Browser behavior

| Variable             | Description                                                                                                                                          | Default                         |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| `KICI_BROWSER_CMD`   | Custom browser command for OAuth login. Supports `{url}` placeholder. Set to `none` to suppress browser opening and print the URL to stdout instead. | Uses the system default browser |
| `KICI_CALLBACK_PORT` | Fixed port for the OAuth PKCE callback server. Useful when firewall rules require a known port.                                                      | Random available port           |

## Development

| Variable            | Description                                                                                                                                                                | Default |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `KICI_DEV`          | Enable development mode. When `true`, pins `@kici-dev/sdk` to the `latest` dist-tag (so a dev registry's prerelease build resolves) and skips npm version resolution.      | unset   |
| `KICI_DEV_REGISTRY` | npm registry the `@kici-dev` scope points at when `KICI_DEV` is set. `kici init` writes it into `.npmrc`; with no value it writes no `.npmrc`.                             | unset   |
| `KICI_DEBUG`        | Enable debug logging. When `true`, prints verbose diagnostics (SDK alias resolution, step-level debug logs, stack traces on errors). Equivalent to the `--debug` CLI flag. | unset   |

## Local dev plane

Read by the [local dev plane](https://docs.kici.dev/user/cli/authoring-and-local/#kici-local) that `kici run <event> --local` dispatches through.

| Variable                           | Description                                                                                                                                                                         | Default              |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `KICI_LOCAL_ORCH_PORT`             | Port the plane orchestrator listens on (HTTP + WebSocket). Change it when another process already holds the default.                                                                | `4319`               |
| `KICI_LOCAL_PG_PORT`               | Port the plane's PostgreSQL listens on.                                                                                                                                             | `45432`              |
| `KICI_LOCAL_PG_MODE`               | Set to `podman` to force the container PostgreSQL fallback instead of the embedded binary.                                                                                          | Embedded PostgreSQL  |
| `KICI_LOCAL_ACCEPTANCE_TIMEOUT_MS` | How long a local run waits for an agent to claim its first job before failing fast. Raise it on a slow host; the run still fails quickly when no scaler label set matches `runsOn`. | `120000` (2 minutes) |

## CI detection

The CLI also reads the conventional CI markers your CI provider sets. They are not KiCI variables — KiCI only consumes them.

| Variable         | Description                                  | Default |
| ---------------- | -------------------------------------------- | ------- |
| `CI`             | Generic CI marker. Set by most CI providers. | unset   |
| `GITHUB_ACTIONS` | Set to `true` by GitHub Actions.             | unset   |
| `GITLAB_CI`      | Set to `true` by GitLab CI/CD.               | unset   |

### How `CI` is interpreted

`kici` treats the environment as CI when `CI`, `GITHUB_ACTIONS`, or `GITLAB_CI` is set to any value other than an explicit opt-out. `0` and `false` are the opt-outs, compared case-insensitively, so `CI=0`, `CI=false`, and `CI=False` all mean "not CI". Surrounding whitespace is ignored, and a value that is empty or only whitespace (`CI=`) is treated as unset.

A vendor marker outranks the generic opt-out: `CI=false GITHUB_ACTIONS=true` is still CI, because an explicit vendor marker names a real runner rather than a preference.

This affects which login flow `kici login` chooses (browser vs device) and whether interactive commands such as `kici init` prompt.

## Usage examples

### CI/CD environment

Authenticate with a pre-existing API key (no browser needed):

```bash
kici login --token "$KICI_API_KEY"
```

### Targeting another environment or custom OIDC provider

`kici login` targets the hosted KiCI Platform by default. To point the CLI at another KiCI environment (e.g. a testing instance) or a custom OIDC provider, override the defaults:

```bash
export KICI_OIDC_ISSUER=https://your-idp.example.com
export KICI_OIDC_CLIENT_ID=your-client-id
export KICI_PLATFORM_URL=https://your-platform.example.com
kici login
```

### Headless SSH session

The CLI auto-detects headless environments and uses the device flow. To force PKCE with URL output instead:

```bash
export KICI_BROWSER_CMD=none
kici login
```

This prints the authorization URL to stdout as `KICI_AUTH_URL=<url>`. Open the URL in any browser to complete authentication.

### Fixed callback port

When behind a firewall or using port forwarding:

```bash
export KICI_CALLBACK_PORT=19876
kici login
```

### Custom config location

Store the KiCI config in a non-default location:

```bash
export KICI_CONFIG_DIR=/tmp/kici-test
kici login
```

---

## Event system

Source: https://docs.kici.dev/user/events/

KiCI supports two broad categories of workflow triggers: **git-based triggers** that work immediately, and **event-based triggers** that use a registration model. Understanding this distinction is key to working effectively with non-git triggers like schedules, custom events, and generic webhooks.

## Overview

Git-based triggers (`push()`, `pr()`, `tag()`, `comment()`, `review()`, `release()`, etc.) work immediately after you commit your lock file. When a GitHub webhook arrives, the orchestrator fetches your lock file and evaluates triggers on the spot -- no advance setup needed.

Event-based triggers work differently. The orchestrator needs to know about them _before_ the event arrives. This is because event-based triggers are matched against a pre-built registration index rather than being evaluated per-event from a lock file fetch. The six event-based trigger types are:

- `kiciEvent()` -- custom events emitted from workflow steps
- `workflowComplete()` -- fires when a workflow finishes
- `jobComplete()` -- fires when a specific job finishes
- `genericWebhook()` -- HTTP webhooks from external services
- `schedule()` -- cron-based time triggers
- `lifecycle()` -- orchestrator lifecycle events (workflow completion, job failure, registration updates)

All six require the **registration model** to function -- covered in detail below.

## Event types

### Custom events

Custom events are user-defined events emitted from workflow steps using `ctx.emit()`. Use `kiciEvent()` to listen for them.

```typescript
import { kiciEvent } from '@kici-dev/sdk';

// Listen for a custom event by name
kiciEvent({ name: 'deploy-complete' });

// With payload matching (JSONPath)
kiciEvent({ name: 'deploy-complete', match: { '$.env': 'prod' } });

// With negative filter
kiciEvent({ name: 'deploy-complete', not: { '$.env': 'staging' } });

// From a specific repository
kiciEvent({ name: 'deploy-complete', source: 'org/infra-repo' });
```

**Config options:** `name` (required), `match`, `not`, `source`, `description`.

### System events

The orchestrator automatically emits completion events when workflows and jobs finish. No manual emission needed -- these fire automatically.

**Workflow completion:**

```typescript
import { workflowComplete } from '@kici-dev/sdk';

// Any workflow completion
workflowComplete();

// Specific workflow by name
workflowComplete({ name: 'build' });

// Only successful completions
workflowComplete({ name: 'build', status: ['success'] });
```

**Config options:** `name`, `status` (`'success'`, `'failed'`, `'cancelled'`), `source`, `description`.

**Job completion:**

```typescript
import { jobComplete } from '@kici-dev/sdk';

// Any job completion
jobComplete();

// Specific workflow + job
jobComplete({ workflow: 'build', job: 'test' });

// Only failures
jobComplete({ workflow: 'build', job: 'test', status: ['failed'] });
```

**Config options:** `workflow`, `job`, `status` (`'success'`, `'failed'`, `'cancelled'`, `'skipped'`), `source`, `description`.

### External events

Generic webhooks let you trigger workflows from any HTTP service -- Stripe, ArgoCD, Slack, Grafana, or your own internal services.

```typescript
import { genericWebhook } from '@kici-dev/sdk';

// Match any event from a source
genericWebhook({ source: 'stripe' });

// Match specific event types
genericWebhook({ source: 'stripe', events: ['invoice.paid'] });

// With HMAC-SHA256 signature verification
genericWebhook({
  source: 'stripe',
  events: ['invoice.paid'],
  auth: {
    method: 'hmac-sha256',
    secret: 'stripe-signing-key',
    signatureHeader: 'stripe-signature',
  },
});

// With API key auth
genericWebhook({
  source: 'slack',
  auth: { method: 'api-key', secret: 'slack-token' },
});
```

**Config options:** `source` (required), `events`, `match`, `not`, `auth`, `path`, `description`.

The `source` field MUST match the `--name` that an operator passed to `kici-admin source add generic --name <name>` when the source was created — that string is the source's identifier in the orchestrator. Generic webhook sources must be created by an operator before events can be received; see [Operator guide: event routing](https://docs.kici.dev/operator/event-routing/) for setup instructions.

### Schedule events

Cron-based triggers evaluated by the orchestrator on a periodic interval. Only the Raft leader evaluates schedules in a clustered deployment.

```typescript
import { schedule } from '@kici-dev/sdk';

// Run every hour
schedule({ cron: '0 * * * *' });

// Run daily at 2 AM UTC
schedule({ cron: '0 2 * * *' });

// Run weekly on Mondays at 9 AM Eastern
schedule({ cron: '0 9 * * 1', timezone: 'America/New_York' });
```

**Config options:** `cron` (required), `timezone` (defaults to `'UTC'`), `description`.

A cron-fired run records the commit sha of the registered lock file as its `sha`. Its `ref` is the repository's default branch, because that is the branch whose lock file the run executes. A [branch restriction](https://docs.kici.dev/user/contexts/#branch-restrictions) matches that branch.

### Lifecycle events

Lifecycle triggers listen for orchestrator-level events related to workflow execution and system state changes.

```typescript
import { lifecycle } from '@kici-dev/sdk';

// Trigger when any workflow completes
lifecycle({ events: ['workflow_complete'] });

// Trigger on job failures from a specific repo
lifecycle({ events: ['job_failed'], sources: ['org/deploy-repo'] });

// Trigger when registrations are updated
lifecycle({ events: ['registration_updated'] });
```

**Available events:** `'workflow_complete'`, `'job_complete'`, `'job_failed'`, `'registration_updated'`.

**Config options:** `events` (required), `sources`, `description`.

## The registration model

This is the most important concept for understanding event-based triggers.

### Why registrations exist

When a GitHub webhook arrives (push, PR, etc.), the orchestrator fetches your lock file from the repository and evaluates triggers on the spot. This works because the event itself tells the orchestrator _which repository_ to look at.

Event-based triggers are different. When a cron timer fires or a custom event is emitted, there is no incoming webhook pointing to a specific repository. The orchestrator needs to know _in advance_ which workflows care about which events. That is what the registration model provides: a pre-built index of event-based workflows.

### How registration works

1. You define a workflow with an event-based trigger (e.g., `schedule()`, `kiciEvent()`, `genericWebhook()`)
2. You compile the workflow (`kici compile`), which produces a lock file
3. You push the lock file to your repository's **default branch** (e.g., `main` or `master`)
4. The orchestrator receives the push webhook, detects it targets the default branch, and extracts all workflows with event-based triggers from the lock file
5. Those workflows are stored in the orchestrator's registration database
6. From that point on, matching events will trigger those workflows

### Key implications

- **Event-based workflows do not trigger until you push to the default branch.** If you add a new `schedule()` workflow, it will not start running until you merge to your default branch. This is by design -- the orchestrator cannot match events to workflows it does not know about.

- **Registration is automatic.** There is no manual setup. Push your code, and the orchestrator handles the rest.

- **Registrations refresh on every default-branch push.** If you add, remove, or modify event-based workflows and push to the default branch, the orchestrator updates its registration index automatically. Removed workflows stop triggering. New workflows start triggering.

- **Git-based triggers are unaffected.** Triggers like `push()`, `pr()`, and `tag()` do not use registrations. They work immediately from any branch because the orchestrator evaluates them per-event from the lock file.

### Practical example

You create a nightly build workflow:

```typescript
import { workflow, job, step, schedule } from '@kici-dev/sdk';

export default workflow('nightly-build', {
  on: schedule({ cron: '0 2 * * *' }),
  jobs: [
    job('build', {
      runsOn: 'linux',
      steps: [
        step('build', async ({ $ }) => {
          await $`pnpm build`;
        }),
      ],
    }),
  ],
});
```

You compile it, commit the lock file, and push to a feature branch. **Nothing happens** -- the cron will not fire because the orchestrator has not registered this workflow yet.

You merge the feature branch into `main`. On the merge push, the orchestrator extracts the `nightly-build` workflow (it has a `ScheduleTrigger`) and registers it. Starting at the next 2 AM UTC, the workflow will trigger.

## How events are matched

When an event arrives, the orchestrator follows this flow:

1. **Event received** -- a custom event is emitted by a step, a cron timer fires, or a generic webhook arrives
2. **Registration lookup** -- the orchestrator queries its registration index for workflows matching the event type (e.g., all workflows with `ScheduleTrigger` for a cron fire, or all workflows with `KiciEventTrigger` for a custom event)
3. **Trigger evaluation** -- for each candidate workflow, the orchestrator evaluates the trigger conditions: event name patterns, payload matching, status filters, source filters
4. **Dispatch** -- matched workflows are dispatched to agents for execution, following the same job queue and agent routing as git-triggered workflows

This lookup is fast because the registration index is held in memory and refreshed only when the registry version changes (on default-branch pushes).

### What an event-triggered run resolves

An event-triggered run takes the same dispatch path as a webhook-triggered run. Four things follow from that.

**Bound contexts resolve in full.** The run reads each job's [contexts](https://docs.kici.dev/user/contexts/): context variables, [scoped secrets](https://docs.kici.dev/user/secrets/), and every protection rule the context carries. A job that calls `ctx.secrets.get()` must bind the context that holds the secret:

```typescript
job('provision', {
  runsOn: ['default'],
  context: 'hetzner-autoscale',
  run: async (ctx) => {
    const token = await ctx.secrets.get('HETZNER_API_TOKEN');
    // ...
  },
});
```

**Protection rules gate the run.** A branch restriction matches the branch the run presents, an approval gate holds it, and a [concurrency group](https://docs.kici.dev/user/concurrency/) serializes it — exactly as for a push or a pull request. A `kiciEvent()` subscriber presents the branch of the run that emitted the event. A scaler event and a failure batch present none, so a branch restriction rejects those two: see [branch restrictions](https://docs.kici.dev/user/contexts/#branch-restrictions). Nothing runs unattended past a gate the operator set. The [approval queue](https://docs.kici.dev/user/dashboard/contexts-and-secrets/#approval-queue) lists each held run, names the context that holds it, and gives the reason. `kici runs show <run-id>` prints the same holds for one run.

**A build job packs the source first.** The run dispatches a `__build__<workflow>` job to an agent labelled `kici:role:builder`, then runs its own jobs against the [cached source and dependency tarballs](https://docs.kici.dev/operator/dependency-caching/). A fleet with no builder-role agent queues that job.

**The run carries a trust tier.** See [trust tiers on internal triggers](https://docs.kici.dev/user/events/#trust-tiers-on-internal-triggers).

### Trust tiers on internal triggers

An internally-triggered run resolves its [trust tier](https://docs.kici.dev/user/contexts/#minimum-trust) from the trigger. The tier decides the run's cache scope, whether it may run a [Dockerfile build](https://docs.kici.dev/user/container-jobs/#who-may-build), whether it receives [install secrets](https://docs.kici.dev/user/private-registries/), and whether a `minimumTrust` context holds it. It also decides whether the run's jobs receive [container-registry credentials](https://docs.kici.dev/user/container-jobs/#private-images) and their declared [git credentials](https://docs.kici.dev/user/patterns/git-credentials/#what-a-job-may-ask-for).

Four rules resolve the tier, and KiCI applies them in this order:

| Order | Trigger                                                                                         | Tier                                                        |
| ----- | ----------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| 1     | A run summoned by an [invoke gate](https://docs.kici.dev/user/global-workflows/#invoking-a-source-repos-own-workflows) | the tier of the summoning run                               |
| 2     | `__schedule_fire`, `kici.scaler.scale-up`, `kici.scaler.scale-down`                             | trusted — no run causes these, the orchestrator mints them  |
| 3     | `__workflows_failed_batch`                                                                      | the most restrictive tier across the failed runs it carries |
| 4     | `__workflow_complete`, `__job_complete`, or a `kiciEvent()` subscriber                          | the tier of the run that emitted the event                  |

Rule 1 runs first on purpose. A workflow author writes the gate's event name, so a gate that named a minted event would otherwise claim rule 2's trusted tier.

Rule 2 lists the three names exactly. A prefix is not enough: the `__` and `kici.` prefixes are both reserved, but only a name on this list is one KiCI mints with no run behind it. Any other reserved name falls through to rule 4, which finds no emitting run and resolves no tier. So an [autoscaling workflow](https://docs.kici.dev/user/workflows/autoscaling-workflows/) that subscribes to `kici.scaler.scale-up` or `kici.scaler.scale-down` runs trusted, and can build a Dockerfile job.

Rule 3 covers the failure batch, which one whole accumulation window of failed runs causes at once. A batch is only as trusted as its least trusted member, so a notifier fired by a window that included one untrusted failure runs at that failure's tier. A window holding more failed runs than the event carries truncates that list to a sample. The batch then resolves no tier at all: a minimum over a sample is not a minimum over the window.

Rule 4 covers the two lifecycle events a single run causes. A run completing does not raise the privilege of what its completion triggers. A `__workflow_complete` subscriber runs at the tier of the run that completed, exactly as a `kiciEvent()` subscriber runs at the tier of the run that emitted.

KiCI resolves no tier it cannot confirm. A missing emitting run, an unreadable tier, or a failed lookup resolves no tier at all, which isolates the run's caches.

An unresolved tier is not uniform across the controls it reaches. The differences are deliberate, and this table is the whole rule:

| Control                | Unresolved tier | A tier below `trusted` |
| ---------------------- | --------------- | ---------------------- |
| Cache scope            | isolated        | isolated               |
| Dockerfile build       | denied          | denied                 |
| Install secrets        | delivered       | stripped               |
| `minimumTrust` context | passes          | holds an `unknown` run |

The first two treat "no tier" as untrusted. The last two treat it as "no opinion", and pass. More than one kind of run carries no resolved tier, and each behaves this way. Among them: an internally-triggered run whose lookup fails, a pull request from a source other than a GitHub App, a cross-source delivery, and a `kici run` remote test run. Only a tier that RESOLVES below `trusted` strips install secrets, and only a tier that resolves `unknown` trips a `minimumTrust` gate.

A subscriber that inherits a tier below `trusted` loses its install secrets. A job that installs from a private registry then fails at install time.

A `minimumTrust` context holds an `unknown` subscriber for security review, whatever value the context declares. Trust is a ref-based judgement with two answers, so `minimumTrust: 'trusted'` and the deprecated `minimumTrust: 'known'` block the same thing. The declared value still decides the wording of the hold reason. A subscriber that inherited the legacy `known` tier from a run row written by an earlier build passes both.

Both symptoms appear far from their cause. The tier belongs to the **emitting** run, so read that run's tier first.

### Cross-source webhook delivery

The catch-all `webhook()` trigger (see [SDK reference: webhook()](https://docs.kici.dev/user/sdk/triggers/#webhook)) participates in this same registration lookup, but with one twist: it fires for matching events arriving via **any** inbound webhook source in the same org, not just the source the workflow's repo is bound to. The orchestrator maintains a `(customerId, eventName)` index over webhook trigger registrations and consults it on every inbound generic webhook.

The lookup is structurally org-isolated — a generic webhook delivered to org A can never reach a workflow registered against org B, because foreign-org rows live in a different bucket of the index. When a webhook fires across sources, the runtime clone token, repo URL, and check-status posting all come from the **registration's** source bundle, not the inbound source. The inbound source contributes only the event payload.

## Circuit breaker

Events can trigger workflows that emit more events, creating chains. The circuit breaker prevents runaway event storms.

### Chain depth limit

Each event carries a `chainDepth` counter. When a workflow triggered by an event emits a new event, the new event's chain depth increments. The orchestrator rejects events that exceed the maximum chain depth.

- **Default limit:** 10 levels deep
- **What happens when hit:** the emission is rejected with a `Circuit breaker tripped` error. The event is never persisted, so it is not queued for later delivery.

For example: Workflow A emits event X (depth 0) -> Workflow B triggers, emits event Y (depth 1) -> ... -> at depth 10, any further emitted events are dropped.

### Rate limiting

Emitted events are rate-limited using a sliding 60-second window, keyed per **(source routing key + event name)** — so one noisy event name in one repository cannot starve the same event name emitted from another.

- **Default limit:** 100 events per (source routing key + event name) per minute
- **What happens when hit:** the emission is rejected with a `Rate limit exceeded` error naming the retry-after delay.
- **System events are exempt:** orchestrator-emitted events (names prefixed `__` or `kici.`) cannot loop, so they bypass the limiter entirely.

Both defaults are configurable. Your operator can set them at startup with `KICI_EVENT_ROUTER_MAX_CHAIN_DEPTH` and `KICI_EVENT_ROUTER_RATE_LIMIT_PER_WORKFLOW_PER_MINUTE` (or the equivalent `eventRouter.maxChainDepth` / `eventRouter.rateLimitPerWorkflowPerMinute` config fields). The rate limit is additionally a live fleet-wide [cluster setting](https://docs.kici.dev/operator/orchestrator/cluster-settings/) — `kici-admin cluster-settings set --event-router-rate-limit-per-workflow-per-minute <n>` takes effect without a restart.

## Delivery guarantees

KiCI's event router delivers every accepted event with **at-least-once** semantics:

- An event that passes the circuit breaker (chain depth + rate limit) and commits
  to the `kici_events` table is guaranteed to dispatch to all matching workflows
  at least once.
- Each dispatch attempt acquires a short-lived lease (default 60 s) on the row.
  If the dispatching node crashes or the handler throws, the lease expires (or
  is released on failure) and the event is automatically retried.
- The retry policy is exponential backoff with full jitter: base 5 s, cap 5 min,
  up to 5 attempts before the event lands in the **DLQ** (dead-letter queue).
  Operators triage DLQ entries via `kici-admin event-dlq list / count / retry / discard`.

**What this means for workflow authors:**

- **Make event handlers idempotent.** A retried dispatch may run a handler more
  than once (e.g. if the first attempt threw after a partial side-effect).
  Workflows that mutate external state should use idempotency keys, conditional
  writes, or other deduplication patterns — same advice as for any distributed
  CI system.
- **Schedule fires are at-least-once too.** A cron schedule that fires while a
  leader is being killed will commit (atomically with `cron_last_fired`) or roll
  back together — never half. Recovery on the new leader does not backfill
  multiple missed instants; if your workflow needs at-least-N guarantees across
  outages, drive it from a different mechanism (e.g. a workflow that runs more
  frequently and emits its own custom event).
- **Drops are still possible — and visible.** Events rejected by the circuit
  breaker (chain depth or rate limit exceeded) are dropped and logged, not
  retried. That's a deliberate safety mechanism; the metric to watch is
  `kici_orch_events_dropped_total{reason}`.

## Emitting custom events

Custom events are emitted from workflow steps using `ctx.emit()`. You can optionally define typed event schemas using `defineEvent()`.

### Basic emission

```typescript
import { workflow, job, step, push } from '@kici-dev/sdk';

export default workflow('build', {
  on: push({ branches: 'main' }),
  jobs: [
    job('build', {
      runsOn: 'linux',
      steps: [
        step('build', async ({ $ }) => {
          await $`pnpm build`;
        }),
        step('notify', async (ctx) => {
          await ctx.emit('build-complete', {
            version: '1.0.0',
            success: true,
          });
        }),
      ],
    }),
  ],
});
```

### Typed event definitions

Use `defineEvent()` with Zod schemas to create a typed contract for event payloads:

```typescript
import { defineEvent, z } from '@kici-dev/sdk';

export const deployComplete = defineEvent(
  'deploy-complete',
  z.object({
    env: z.string(),
    version: z.string(),
    services: z.array(z.string()),
  }),
);
```

Then emit using the definition — the payload is checked against the schema:

```typescript
step('emit', async (ctx) => {
  await ctx.emit(deployComplete, {
    env: 'prod',
    version: '1.2.3',
    services: ['api', 'web'],
  });
});
```

And consume in another workflow:

```typescript
import { workflow, job, step, kiciEvent } from '@kici-dev/sdk';

export default workflow('post-deploy', {
  on: kiciEvent({ name: 'deploy-complete', match: { '$.env': 'prod' } }),
  jobs: [
    job('smoke-test', {
      runsOn: 'linux',
      steps: [
        step('test', async ({ $ }) => {
          await $`./scripts/smoke-test.sh`;
        }),
      ],
    }),
  ],
});
```

Custom events are delivered immediately when emitted (mid-workflow, not queued until workflow completion). See the [SDK reference: emitting events](https://docs.kici.dev/user/sdk/validation-events/#emitting-events) section for the full `ctx.emit()` API.

### Reserved event names

Two name prefixes belong to the orchestrator, and `ctx.emit()` refuses both:

- `__` -- the orchestrator's own lifecycle and schedule events (`__schedule_fire`, `__workflow_complete`, `__job_complete`, `__workflows_failed_batch`).
- `kici.` -- KiCI internal system events.

A step that emits either name fails with `event name prefix "__" is reserved for KiCI internal events and cannot be emitted from a workflow step (got "__foo")`, naming the prefix that matched. A caller that reaches the orchestrator without the SDK gets the shorter `event name prefix "__" is reserved for KiCI internal events`, and no event is written. These events run at a higher trust level and skip the rate limiter, so a workflow must not be able to forge one. Prefix only -- a name that merely contains the text, such as `deploy__done`, is fine.

The same reservation covers an [invoke gate](https://docs.kici.dev/user/global-workflows/#invoking-a-source-repos-own-workflows). `invokeSource()` rejects a reserved name when you compile. A lock file that still carries one fails the gate job at dispatch, with status `failed` rather than skipped.

Subscribing is unaffected: `kiciEvent({ name })` may name a reserved event, and only emission is refused.

## See also

- [SDK reference: event triggers](https://docs.kici.dev/user/sdk/triggers/#event-triggers) -- complete API signatures for all trigger builders
- [SDK reference: emitting events](https://docs.kici.dev/user/sdk/validation-events/#emitting-events) -- `ctx.emit()` and `defineEvent()` API
- [Workflow patterns: workflow chaining](https://docs.kici.dev/user/patterns/integrations/#workflow-chaining) -- examples of event-driven workflow chains
- [Operator guide: event routing](https://docs.kici.dev/operator/event-routing/) -- configuring generic webhook sources, trust relationships, and event routing
- [Architecture: event system](https://docs.kici.dev/architecture/webhooks/event-system/) -- internal event routing design, registration model, cluster synchronization

---

## Global workflows

Source: https://docs.kici.dev/user/global-workflows/

Global workflows let one **workflow repo** define jobs that run on events from many **source repos** in the same org. They're the answer to "I want one CI policy / release pipeline / security scan to fire on every repo without copy-pasting `.kici/` folders everywhere."

If you've only ever used per-repo workflows so far, start with the mental model section — global workflows add two new concepts (workflow repo vs. source repo, and authoring vs. source axes) that show up everywhere from SDK syntax to dashboard settings.

## Mental model

| Term           | Meaning                                                                                                                                                |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Workflow repo  | The repo whose `.kici/workflows/*.ts` file **declares** the global workflow. Holds the steps. Also known as the _authoring_ repo.                      |
| Source repo    | The repo that **emits** the event (push / PR / tag / ...) that causes the global workflow to fire. The agent checks out this repo as the working copy. |
| Global         | A workflow whose trigger carries one or more `repos:` glob patterns. The presence of `repos:` is what classifies a workflow as global.                 |
| Authoring axis | Policy that answers "which repos may **author** global workflows?" Controlled by the allow-list in the dashboard's _Allowed author repos_ setting.     |
| Source axis    | Policy that answers "which **source** repos' events are allowed to trigger global workflows?" Controlled by the deny-list in _Blocked source repos_.   |

The two axes are independent. A global workflow fires only if it passes **both** — its authoring repo is allowed AND the source repo is not denied.

## Declaring a global workflow

Add `repos:` to any trigger. Any workflow with at least one `repos:`-bearing trigger becomes global automatically; no separate flag is required.

```ts
import { workflow, job, step, push } from '@kici-dev/sdk';

export default workflow('org-lint', {
  on: [
    push({
      repos: ['myorg/*', '!myorg/archived-*'],
      branches: ['main'],
    }),
  ],
  jobs: [
    job('lint', {
      steps: [
        step('lint-all', async ({ $, env }) => {
          await $`echo source=${env.KICI_SOURCE_REPO_PATH ?? 'unknown'}`;
          await $`npm run lint`;
        }),
      ],
    }),
  ],
});
```

Patterns in `repos:` use the same globbing as `branches:` / `paths:` — plain globs (`myorg/*`), a leading `!` for exclusions (`!myorg/fork-*`), and a fully-qualified `owner/repo` identity for exact matches (`myorg/platform`). A bare `**` matches every repo in the org, including one whose identifier starts with a dot (`.github/workflows-config`) — a repo identifier is an owner/name pair, not a file path, so a leading dot carries no meaning of its own. Path globs in `paths:` keep the usual convention and do not match dot-prefixed files unless the pattern spells the dot out.

### At a dual-repo checkout

The agent checks out both repos. **Inside a step body**, `env` carries a pointer to each working tree:

| `env` var                 | Carries                                                                                                                                      |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `KICI_SOURCE_REPO_PATH`   | The **source** repo's working tree (the repo that emitted the event). This is the repo the job's `$` / `git` commands operate on by default. |
| `KICI_WORKFLOW_REPO_PATH` | The **workflow** repo's working tree (the repo that authored the workflow). Useful for reading shared scripts or config from your CI repo.   |
| `KICI_SOURCE_REPO`        | The source repo's `owner/repo` identifier — the same value as `ctx.event.sourceRepo`.                                                        |
| `KICI_WORKFLOW_REPO`      | The workflow repo's `owner/repo` identifier.                                                                                                 |
| `KICI_SOURCE_BRANCH`      | The source repo's checked-out ref. **Empty string** when the event carries no single ref.                                                    |
| `KICI_SOURCE_SHA`         | The source repo's checked-out commit. **Empty string** when the event carries no single sha.                                                 |
| `KICI_IS_GLOBAL_WORKFLOW` | `"true"`. Never set on the same-repo path, so it is the cheapest test for which path you are on.                                             |

All seven are set **only when there are two repos to point at**. An event from the workflow's own repo is matched from that repo's lock file, not as a global candidate. The workflow then runs as an ordinary single-repo workflow: one checkout, and none of the seven set. Read them with a fallback, as the example above does.

Guard those two on emptiness rather than absence: `??` does not catch `""`, but `||` does.

These are real process environment variables for the whole job, so a subprocess a step spawns inherits them: `` await $`echo $KICI_SOURCE_REPO` `` works. What does **not** see them is anything resolved outside that process — a job-level `env:` block or a container image's entrypoint, both of which are settled before the job starts. Outside a step body, use the `sourceRepo` / `workflowRepo` pair on the filter, generator, and rule contexts described below.

A global workflow's job runs with **no secrets at all** — neither the source repo's nor its own. See _Secrets are not available_ below.

### The triggering event

`ctx.event` inside a global workflow's job is the **source** repo's normalized event — the push or PR that fired the workflow, from a repo the workflow's own author may not own. `ctx.event.sourceRepo` names that repo.

That field is what makes a per-source-repo concurrency group expressible — and you have to write it. A global workflow runs on events from many repos, and their default branches share a name, so a group keyed on the branch alone puts every repo in one group, and with `cancelInProgress` (the default) one repo's push cancels another repo's in-flight run. That is still the behaviour of a branch-only group; naming the source repo in the key is what separates them:

```ts
concurrency: {
  group: ({ branch, event }) => `${event.sourceRepo}:${branch}`,
  cancelInProgress: true,
},
```

### Narrowing to the repos that need it

A global workflow that matches `myorg/*` will, by default, run on every repo in the org. Three mechanisms narrow it to the repos it actually applies to, in increasing order of power:

1. **A `requires` content filter on the trigger** — the cheapest gate. The orchestrator checks a file's contents (a JSON-path probe over `package.json`, for example) and drops the workflow **before any agent is dispatched** when the condition is not met. See [`requires` on triggers](https://docs.kici.dev/user/sdk/triggers/#content-requirements-requires). This is provider-dependent — it needs a file-contents fetcher, which the GitHub provider supplies.
2. **A workflow-level `filter` predicate** — arbitrary TypeScript over the checked-out source tree (below). Works with any provider that clones.
3. **A `DynamicJobFn`** — generate the exact job set from the source repo's state ([Generating jobs per source repo](https://docs.kici.dev/user/global-workflows/#generating-jobs-per-source-repo) below).

### Narrowing with a filter

Before reaching for a `filter`, check whether a declarative filter answers the question. `commitMessage` (on the trigger) and `requires` (over source files) are evaluated by the orchestrator from data it already has, so they cost no evaluation job at all — while a `filter` predicate dispatches one per (event × workflow repo). Gating on a `[skip ci]` marker, a conventional-commit prefix, or the contents of a named config file needs no predicate.

A workflow can declare a `filter`: a predicate that decides whether the workflow applies to this event at all.

```ts
import { workflow, job, step, push } from '@kici-dev/sdk';

export default workflow('org-container-lint', {
  on: [push({ repos: ['myorg/*'] })],
  filter: async ({ sourceRepo, changedFilesStatus, $ }) => {
    // `changedFiles` throws when the diff is unavailable, so guard first.
    if (changedFilesStatus !== 'fetched') return true;
    const found = await $`ls ${sourceRepo.path}`;
    return found.stdout.includes('Dockerfile');
  },
  jobs: [
    job('lint-dockerfile', {
      runsOn: ['kici:os:linux'],
      steps: [
        step('lint', async ({ $, env }) => $`hadolint ${env.KICI_SOURCE_REPO_PATH}/Dockerfile`),
      ],
    }),
  ],
});
```

The filter receives a `FilterContext`:

| Property             | Type                                      | Description                                                                                |
| -------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------ |
| `sourceRepo`         | `RepoInfo`                                | The repo whose event triggered this evaluation, checked out on the evaluating agent.       |
| `workflowRepo`       | `RepoInfo`                                | The repo that registered the workflow. Identical to `sourceRepo` for a same-repo workflow. |
| `event`              | `EventPayload`                            | The normalized event envelope.                                                             |
| `changedFiles`       | `string[]`                                | Files changed in this event. Throws when unavailable — guard with `changedFilesStatus`.    |
| `changedFilesStatus` | `'fetched' \| 'unavailable' \| 'skipped'` | Whether `changedFiles` can be read.                                                        |
| `env`                | `Record<string, string\|undefined>`       | Environment variables.                                                                     |
| `$`                  | zx shell                                  | Shell executor.                                                                            |

`RepoInfo` carries `path` (an absolute path to the checkout on the evaluating agent) plus optional `ref` and `sha`. **Both are optional** — an event that carries no single ref leaves them undefined, so guard before reading them.

**`sourceRepo.path` is not stable across evaluations.** Its _contents_ are: the evaluating agent and the later run see the same tree at the same commit. The path itself is not — a different working directory, and possibly a different machine. Read _through_ it; never embed it in a job name, an output, or anything compared across calls.

**A `filter` must be pure and deterministic.** Decide from the context alone — the event, the changed files, and the checked-out tree — so the same event always yields the same verdict.

### Global and same-repo filters differ

The same `filter` keyword means two different things depending on whether the workflow is global:

|                                | Global workflow (`repos:` on a trigger)          | Same-repo workflow                                                        |
| ------------------------------ | ------------------------------------------------ | ------------------------------------------------------------------------- |
| Evaluated                      | once per (event × workflow repo)                 | once per job that reaches dispatch, and once per job generator            |
| Evaluated relative to the run  | **before** any run row exists                    | **after** the run row exists                                              |
| A `false` verdict leaves       | no run at all — nothing appears in the dashboard | a run whose only entries are the evaluation jobs, rolling up to `success` |
| `sourceRepo` vs `workflowRepo` | two different repos                              | the same repo                                                             |

Two consequences of the same-repo shape are worth designing for. A workflow with ten jobs calls its filter ten times for one event — each on its own agent with its own checkout and its own `$` — so anything the predicate does happens that many times: keep it cheap and side-effect free. And if the predicate can answer differently for the same event, the workflow will _partially_ dispatch, running some jobs and not others.

**A held or rejected job is not filtered at all.** A job held for approval, or rejected by a context rule, already has a gate — the hold or the rule — so it never takes a filter verdict, and an approved job dispatches without one. Concretely: a path filter cannot stop an approval request for a job the change does not concern.

### Generating jobs per source repo

A global workflow's job generators run in the same pre-run evaluation as the filter, with both repos on disk. `sourceRepo` and `workflowRepo` are on the generator context, so one workflow repo can produce a different job set per source repo:

```ts
import { job, step, workflow, push, type DynamicJobFn } from '@kici-dev/sdk';
import { readFile } from 'node:fs/promises';

const perRepoJobs: DynamicJobFn = async ({ sourceRepo }) => {
  if (!sourceRepo) return [];
  const pkg = JSON.parse(await readFile(`${sourceRepo.path}/package.json`, 'utf8'));
  return Object.keys(pkg.scripts ?? {})
    .filter((s) => s.startsWith('ci:'))
    .map((s) =>
      job(s.replace(':', '-'), {
        runsOn: ['kici:os:linux'],
        steps: [step('run', async ({ $ }) => $`pnpm ${s}`)],
      }),
    );
};

export default workflow('org-ci', {
  on: [push({ repos: ['myorg/*'] })],
  jobs: [perRepoJobs],
});
```

The same `sourceRepo.path` caution applies: read the tree through it, and derive job names from the repo's _contents_, never from the path.

## Invoking a source repo's own workflows

A global workflow can run the source repo's **own** workflows and gate on them. Use the `invoke:` job option, built with `invokeSource()`:

```ts
import { job, workflow, push, kiciEvent, invokeSource } from '@kici-dev/sdk';

// Source repo (myorg/backend/.kici/workflows/tests.ts) — opts in by subscribing.
export const repoTests = workflow('repo-tests', {
  on: [kiciEvent({ name: 'myorg.repo-tests' })],
  jobs: [
    job('unit', {
      runsOn: ['kici:os:linux'],
      run: async ({ $ }) => {
        await $`npm test`;
      },
    }),
  ],
});

// Global workflow (myorg/ci-pipelines/.kici/workflows/org-pipeline.ts).
export default workflow('org-pipeline', {
  on: [push({ repos: ['myorg/*'], branches: ['main'] })],
  jobs: [
    // The invoke gate: emits `myorg.repo-tests` at the source repo and waits for
    // every run it triggers. It runs no steps of its own.
    job('repo-tests', { invoke: invokeSource('myorg.repo-tests') }),

    // Gated on the invoked runs through the standard needs vocabulary.
    job('deploy', {
      needs: ['repo-tests'],
      runsOn: ['kici:os:linux'],
      run: async (ctx) => {
        for (const r of ctx.needs['repo-tests'].result) {
          // r = { repo, workflow, runId, status, outputs }
          if (r.status === 'success') ctx.log.info(`coverage=${r.outputs.coverage}`);
        }
      },
    }),
  ],
});
```

An invoke gate never runs steps, so it is mutually exclusive with `steps` / `run`. A repo opts in by subscribing to the event with `kiciEvent({ name })` — a global cannot invoke a repo that did not subscribe.

The gate's event name follows the same rule as `ctx.emit`: the prefixes `__` and `kici.` are reserved for KiCI, and `invokeSource()` rejects them when you compile. A lock file that still carries a reserved gate fails that gate job at dispatch with `invoke gate cannot summon '…': the event-name prefix "…" is reserved for KiCI internal events. Choose a name a workflow may emit.` The job's status is `failed`, not skipped, so `optional` does not turn the refusal green. Nothing is summoned and no proxy job is created. See [reserved event names](https://docs.kici.dev/user/events/#reserved-event-names).

A summoned run inherits the [trust tier](https://docs.kici.dev/user/events/#trust-tiers-on-internal-triggers) of the run that holds the gate.

### Required by default

An emit that matches **zero** subscribers **fails** the gate. A repo that forgot to wire up its tests must not silently pass the org gate. To let a repo opt out, pass `optional`:

```ts
job('repo-tests', { invoke: invokeSource('myorg.repo-tests', { optional: true }) });
```

A zero-subscriber gate with `optional: true` succeeds immediately with no proxies. `optional` is separate from `continueOnError`: `optional` governs whether there was anything to invoke, `continueOnError` governs whether an invoked run passed.

### Reading invoked-run results

Each invoked run appears as a **proxy node** under the gate in the run graph, and its result is available to downstream jobs on `ctx.needs['<gate>'].result` — an array of `{ repo, workflow, runId, status, outputs }`, one entry per invoked run. `outputs` carries the run's non-secret declared outputs; a repo's secret outputs never cross into the global run.

### Standard job options apply

The gate is a standard job. Tolerate a failed invoked run with `continueOnError`, react to a failed gate with a downstream `needs` `when: 'on-failure'`, bound the wait with the job `timeout`, and bound the fan-out with `maxParallel` / `failFast`:

```ts
job('repo-tests', {
  invoke: invokeSource('myorg.repo-tests'),
  continueOnError: true,
  timeout: '1h',
  maxParallel: 10,
  failFast: true,
});
```

### Generating invoke gates

Because `invoke:` is a job shape, a generator can inspect the source repo and return only the gates that apply:

```ts
import { existsSync } from 'node:fs';
import { join } from 'node:path';

const perRepoGates: DynamicJobFn = async ({ sourceRepo }) => {
  if (!sourceRepo) return [];
  const jobs = [];
  if (existsSync(join(sourceRepo.path, 'Dockerfile')))
    jobs.push(job('docker', { invoke: invokeSource('myorg.docker-test', { optional: true }) }));
  if (existsSync(join(sourceRepo.path, 'package.json')))
    jobs.push(job('node', { invoke: invokeSource('myorg.node-test') }));
  return jobs;
};
```

The generator decides whether to create a gate at all; `optional` decides what a created gate does when nothing subscribes.

**Set a `timeout` when a summoned run can be held.** A gate waits for every run it summoned. An invoked run binds its own [contexts](https://docs.kici.dev/user/contexts/), so a [protection rule](https://docs.kici.dev/user/contexts/#protection-rules) can hold it for reviewer approval or a wait timer. A gate with no `timeout` then waits for as long as the hold lasts, which is until a human acts on it. Give such a gate a `timeout` so the wait is bounded.

## Enabling global workflows

Global workflows are gated by a **fleet-wide master switch** held by the orchestrator operator, off by default. Until it is on, `repos:`-bearing workflows are registered but never dispatched.

1. **The operator enables it cluster-wide** with `kici-admin cluster-settings set --global-workflows-enabled true`. This is the kill-switch — every per-org control below is ignored while it is off, and it cannot be flipped from the dashboard. The dashboard's **Settings → Global workflows** tab shows its current state as a read-only badge.
2. In the dashboard → **Settings → Global workflows**, decide which authoring/source controls you need. These per-org lists stay dashboard-editable; an org that has set none means "no per-org restrictions", not a denial.

| Setting              | What it controls                                                                                                                                                                                     | Typical use                                                                            |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| Allowed author repos | Restricts which repos can **author** (register) global workflows. Globs matched against the authoring repo identifier. When OFF, any repo in the org may author globals.                             | Lock authoring to `myorg/ci-*` so random product repos can't ship org-wide automation. |
| Blocked source repos | Blocks dispatch for events emitted from these **source** repos, regardless of authoring. Globs matched against the event source repo identifier. When OFF, events from any repo may trigger globals. | Protect against fork spam — e.g. `!myorg/*` via `myorg/fork-*`.                        |
| Elevated access      | **Deprecated and not enforced.** Stored and echoed back, but nothing reads it — a global workflow's job receives no secrets, so there is no access for it to grant. See _Secrets are not available_. | None. Clear the list so it does not imply a grant that is not in force.                |

All three lists accept globs. Leading `!` inside a single pattern is not supported here; negation is via the list-is-implicit-deny semantics, so keep it simple (`myorg/ci-*`, `myorg/platform-*`).

Patterns match repo identifiers by the same rule as `repos:` on a trigger: an identifier is an owner/name pair, not a file path, so a leading dot carries no meaning of its own and a wildcard segment matches one. `myorg/*` covers `myorg/.github`, and `**` covers every repo in the org. Review any existing entry that relies on a wildcard to reach — or to spare — a dot-prefixed repo name.

### Saving and reverting

The page is a two-state editor — changes are local until you click **Save changes**, and you can abandon them with **Discard changes**. There is no partial save; the PATCH is all-or-nothing per save click.

## Security model

### Two independent axes

A global workflow fires only if:

1. **The authoring repo is allowed.** If _Allowed author repos_ is ON, the workflow's authoring repo must match at least one allow-list glob. If OFF, any repo may author. Enforced at two points:
   - At registration time (extraction from the lock file — non-matching globals are dropped, and the orchestrator logs `Global workflows excluded from registration` naming each one).
   - At dispatch time (defense-in-depth — policy changes after registration still take effect).
2. **The source repo is not denied.** If the event's source repo matches any glob in _Blocked source repos_, the global workflow is skipped. Enforced at dispatch time.

Both checks are logged to the orchestrator. Grep for `Global workflows excluded from registration` (registration time) and `Skipping global workflow dispatch` (dispatch time) to see enforcement in action.

Both checks read the settings of the organization the **event's source** resolves to. If no webhook source maps the event's routing key to an organization, the orchestrator resolves the built-in `__default__` organization anchor instead. That anchor is an ordinary organization for policy purposes: it carries no per-org lists, so by the empty-list rule above it restricts nothing, and the fleet-wide master switch alone governs it. A deployment whose sources are unmapped — the state a fresh install starts in, since the quickstart configures no sources — runs global workflows normally once that switch is on.

Map a source to a real organization when you want per-org policy to be **expressible**: `kici-admin source update <routingKey> --customer-id <org>`. Allow- and deny-lists are stored per organization, so every unmapped source shares the one policy surface on the `__default__` anchor. That is a reason to map, not a precondition for dispatch.

The registration log line names the organization it decided against, so a refusal is always attributable to a specific policy rather than to the anchor itself. See the troubleshooting table below.

### Secrets are not available

A global workflow's job is dispatched with **no secret material** — not the source repo's, and not the workflow repo's own. The organization-wide dispatch path binds no secret contexts, so a `contexts:` declaration on a global workflow resolves to nothing and any secret the steps expect is absent. Plan for it: a global workflow is for checks, policy and reporting that need only the two checkouts, not for deploys that need credentials.

This is about your **stored secrets**, not about repository access: the job is still handed a short-lived clone token for each repo it checks out, which is how the dual checkout works at all. What it does not get is anything from a secret context.

To run something that needs secrets on a source repo's event, put those jobs in a per-repository workflow in that repo, where the workflow's `contexts:` resolve normally.

The **Elevated access** setting reads as the way to lift this, and it is not: it is **deprecated and never consulted**. Nothing in the dispatch path reads the list, and adding a repo to it does not make any secret readable. It is kept only so an existing value stays visible and clearable, and is removed at the next major version — see [Deprecations](https://docs.kici.dev/user/deprecations/).

## When does it fire?

Same-repo globals (a workflow in `myorg/app` with `repos: ['myorg/app']`) fire on pushes to `myorg/app`. Cross-repo globals fire on pushes to any source repo whose identifier matches a glob on the authoring workflow's trigger. The orchestrator de-duplicates between the per-repo and cross-repo matching passes, so a single event produces at most one run per (workflow, source-repo, trigger) triple.

Non-push triggers work too — `pr()`, `tag()`, `comment()`, `release()`, `workflowRun()`, etc. all accept `repos:`. `kiciEvent()` / `schedule()` / cron-like triggers have no source repo, so they're always per-org-registered regardless of `repos:`.

A global workflow that declares a `filter` or a job generator is decided by one **evaluation job per (event × workflow repo)**, dispatched before any run exists. That job checks out both repos once and evaluates every candidate workflow from that repo, so ten global workflows in one CI repo cost one evaluation, not ten.

When that evaluation cannot reach a verdict — it fails, breaches its budget, or never reports — the workflows it was deciding on **do not run**. On a provider that supports commit checks, that posts a `failure` check named **`KiCI: Organization workflow evaluation`** on the source commit, so the outcome is visible instead of silent. Three things to know about it:

- The check is posted whether the evaluation failed **outright** or only **partly**. A per-workflow budget breach, or a `filter` that throws, leaves that one workflow undecided while its neighbours from the same repo are decided and run normally; the check then names only the undecided ones. So a broken `filter` is reported the same way whether or not other global workflows happen to share its repo.
- Branch protection that lists required checks by name is unaffected, because the check is not on that list. Merge automation that requires _every_ check to be green will block on it.
- **Re-run the failed evaluation to clear the check.** A failed evaluation is recorded as one errored run named `__globaleval__<owner>/<workflow-repo>`. Fix the cause, then re-run that run — `kici runs rerun <run-id>`, or the **Re-run** button on the run in the dashboard. The re-run re-evaluates the original event against the workflow repo's current state, dispatches whatever it now admits, and posts a `success` check under the same name on the same commit. The request is **accepted immediately**; the evaluation itself is a job on an agent and runs after the answer, exactly as it does for the push that first triggered it. So watch the run and the check for the outcome, not the response. The check clears only when the re-evaluation reaches a verdict: if it fails again, or the orchestrator cannot run it, the `failure` check stands. A provider redelivery of the same webhook will not do this: it is dropped as a duplicate. Pushing a new commit also works, and is what you need when the payload of the original delivery is no longer stored.
- **Two failed evaluations on one commit share the check.** The check name carries no repo, so if two workflow repos both fail on the same push, re-running one of them posts `success` over the other's `failure`. The success summary names the workflow repo it re-evaluated; re-run the other round too.

## Approval gates are not supported

A global workflow cannot carry an `approval` gate, at the workflow level or on a job. Approval holds are applied by the per-repository dispatch path; the global path dispatches its jobs without consulting one, so a gate declared here would never be enforced. `kici compile` refuses it with `error [E124]` rather than accepting a security control the workflow does not actually have. A job produced by a `dynamicJob` generator never passes through the compiler, so that case is caught at dispatch instead — the orchestrator logs an error naming the workflow and job, and runs it ungated.

To gate a deployment behind a human, put the gated jobs in a workflow whose triggers carry no `repos:`.

## Re-running an organization-wide run

An organization-wide run that executed against another repository cannot be re-run from that repository. This is a permanent authorization boundary, not a limitation.

The re-run path resolves a workflow out of the repo the run acted on. For an organization-wide run that is the **source** repo, not the workflow repo that declares it. So a re-run from the source repo would re-execute the defining repo's code without the defining repo's policy pass. If the source repo carries a workflow of the same name, the re-run would run that workflow instead — with the source repo's credentials and none of the organization-wide job configuration.

Two tiers refuse it: your orchestrator, and the hosted Platform on every path that exposes re-run. Each refusal names both repos.

That refusal is what makes the run visible to both teams. A member scoped to **either** repo reads and cancels the run. Neither team can re-execute the other's code.

To run it again, trigger it from the repo that defines the workflow. You can also push a new commit to the source repo; a provider redelivery of the same event is dropped as a duplicate.

A failed organization-workflow **evaluation** is the exception. Re-running one re-evaluates the original event instead of resolving a workflow, so the substitution above cannot happen. See [When does it fire?](https://docs.kici.dev/user/global-workflows/#when-does-it-fire).

## Notifications

An organization-wide run belongs to two repositories: the one it executed against, and the one that defines the workflow. A notification subscription's repository filter matches on either. So a subscription scoped to the defining repo hears about every organization-wide run of its own workflows, even though those runs execute against other repositories.

This is the same either-repo rule the run history uses, so the runs a team sees in the dashboard are the runs it is notified about.

## Requirements a filter places on the run

A `filter` reads the source tree, so the evaluation must be able to obtain one. A job that restores its workflow source from the cache and has no source repository to clone from fails with an explicit error rather than evaluating the filter against an empty tree. This applies to dispatch paths that run without a source repository configured — a filter and such a path are mutually exclusive; drop one or the other.

## Troubleshooting

| Symptom                                                                                         | Likely cause                                                                                                                                                                                                           | Where to look                                                                                                                                                                                                                                                                                                                                |
| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Global workflow registered but never runs                                                       | Master toggle OFF, or allow-list blocks the authoring repo, or deny-list blocks the source repo                                                                                                                        | Orchestrator log: `Skipping global workflow dispatch` (dispatch time) / `Global workflows excluded from registration` (registration time)                                                                                                                                                                                                    |
| A global workflow is never registered at all — it is absent from `kici-admin registration list` | The fleet-wide master switch is off, or the authoring repo does not match a populated _Allowed author repos_ list.                                                                                                     | Orchestrator log: `Global workflows excluded from registration`, naming the organization it decided against. Check the switch first (`kici-admin cluster-settings show`), then that org's allow-list in the dashboard. An `"orgId": "__default__"` in the line is not itself the fault — that anchor carries no lists and restricts nothing. |
| `repos:` has no effect — workflow only fires on its own repo                                    | The fleet-wide master switch is off. Without it, the orchestrator treats the workflow as per-repo-only.                                                                                                                | Check the fleet-wide switch with `kici-admin cluster-settings show`. The dashboard → Settings → Global workflows tab shows it as a read-only badge.                                                                                                                                                                                          |
| Secrets unavailable in a global job                                                             | Expected — a global workflow's job receives no secrets at all, and the _Elevated access_ list is not enforced.                                                                                                         | Move the jobs that need credentials into a per-repository workflow in the repo that owns the secrets                                                                                                                                                                                                                                         |
| Dashboard shows workflow twice after registering                                                | Both a generic webhook source and a provider source (github, generic) re-registered the same repo.                                                                                                                     | Check `workflow_registrations` via `kici-admin workflow list` and confirm the right routing key owns the workflow.                                                                                                                                                                                                                           |
| Global workflow registered, enabled, allowed — and still no run appears                         | Its `filter` returned `false`. A global filter runs before the run is created, so a suppressed workflow leaves nothing behind at all.                                                                                  | [Reading a global workflow's filter output](https://docs.kici.dev/user/global-workflows/#reading-a-global-workflows-filter-output) — the evaluation round's own log. The orchestrator also logs `Global workflow skipped by eval round`, naming the workflow and the reason.                                                                                                             |
| Global workflow never fires for one particular source repo                                      | Its `repos:` patterns do not match that repo's identifier.                                                                                                                                                             | Orchestrator log: `Global workflows dropped by their repos filter` — one line per delivery, naming each dropped workflow, its repo and its patterns.                                                                                                                                                                                         |
| A `failure` check named `KiCI: Organization workflow evaluation` on a commit                    | The pre-run evaluation failed or timed out, so the global workflows from that repo were not run.                                                                                                                       | Orchestrator log for the evaluation job. Fix the cause, then re-run the errored `__globaleval__…` run (`kici runs rerun <run-id>`) to re-evaluate and clear the check; a redelivery is dropped as a duplicate.                                                                                                                               |
| Same-repo workflow shows a `success` run with no jobs in it                                     | Its `filter` returned `false`. A same-repo filter runs after the run exists, so the run remains, carrying only the evaluation jobs.                                                                                    | The run detail page — the evaluation job's log records the filter verdict.                                                                                                                                                                                                                                                                   |
| Re-run is refused with "Cannot re-run an organization-wide workflow"                            | Expected — the run executed against a source repo that does not declare the workflow.                                                                                                                                  | [Re-running an organization-wide run](https://docs.kici.dev/user/global-workflows/#re-running-an-organization-wide-run) — trigger it from the repo that defines the workflow instead.                                                                                                                                                                                                    |
| Every global workflow stopped running right after an orchestrator upgrade                       | The agents were not upgraded first. An agent older than v0.5.0 cannot evaluate a global workflow, and one containing a `dynamicJob` now needs an evaluation even without a `filter` — so its **static** jobs stop too. | The `KiCI: Organization workflow evaluation` check names the agent versions it found. Upgrade every `kici:role:init-runner` agent to v0.5.0 or newer.                                                                                                                                                                                        |

### Reading the decision trace for a delivery

The dashboard records why each workflow did or did not fire. Open
**Settings → Event log**, select the delivery, and read the **Workflow
decisions** section. It lists every workflow the delivery was evaluated
against — per-repository and organization-wide alike.

Each entry expands to the individual checks the trigger evaluation performed:
the check, the pattern, the value tested against it, and whether it passed. An
organization-wide workflow is named with the repository that defines it, so you
can find your own workflow even though it is absent from the source
repository's lock file.

This is the first place to look when a workflow does not fire. A failed `repo`
check means the `repos:` patterns do not match the source repository. A failed
`filter` check means the evaluation round excluded the workflow.

The value a check tested and the reason it gives quote the webhook body, so
those two fields need the `event_log:read_payload` permission. Without it the
row still names the check, the pattern, and whether it passed.

The orchestrator records the trace when trigger matching runs. A delivery the
Platform rejected at the relay therefore has none.

### Reading a global run in the dashboard

A global run is attributed to the **source** repo — the repo whose event
triggered it, and whose code the jobs check out. Its run detail page names both
repos, so you can tell it apart from an ordinary per-repo run:

| Row          | Shows                                                                    |
| ------------ | ------------------------------------------------------------------------ |
| `Repository` | the source repo — the one the run acted on                               |
| `Defined in` | the workflow repo, tagged `Organization-wide`. Absent on an ordinary run |
| `Workflow`   | links into the **workflow** repo, on its default branch                  |

The `Workflow` link points at the workflow repo's default branch rather than at
a commit: the run's own commit belongs to the source repo, and nothing records
which commit of the workflow repo a given run used. So the link always shows the
file as it stands now, which may have changed since the run.

The `Payload` tab shows the source repo's event — the webhook delivery the
workflow reacted to, which for a global workflow comes from a repo you may not
own. A global run dispatched before your orchestrator stored payloads for this
path has none, and its tab reports that it could not load one.

#### Who can see it

A global run belongs to **both** repos, so a member whose role is scoped to
either one reaches it — the team whose push triggered it, and the team that
authored the workflow. Both see it in the run list, in the repository filter
(which offers both names), and on the run detail page. Cancelling follows the
same rule, so the team whose workflow is running can always stop it.

Releasing a **held** run is the one exception: approving a hold permits code to
run against the source repo, so it stays with a member scoped to that repo. A
member scoped only to the workflow repo sees the run but not its hold.

This applies only where the two repos genuinely differ. An ordinary per-repo run
records no separate workflow repo and is scoped to its own repo exactly as
before, and a member scoped to neither repo sees nothing in either case.

### Reading a global workflow's filter output

A global workflow's `filter` runs in a pre-run evaluation round, and that round
decides whether a run exists at all — so on the path where it suppresses a
workflow there is no run, and nothing appears in the dashboard. The round's own
log is still recorded. Read it with the orchestrator admin CLI, in two steps:

```bash
# 1. Find the round. Its workflow name is __globaleval__<owner>/<repo> of the
#    WORKFLOW repo. In the JSON rows, `id` is the job id and `run_id` is the
#    run id.
kici-admin queue list --workflow-name '__globaleval__myorg/ci-pipelines' --limit 5 --json

# 2. Print the round's log (step 0 is the evaluation itself).
kici-admin runs logs <run_id> --job <id>
```

Use `--json` on the first command: the plain table abbreviates both ids to their
first eight characters, and the second command needs them in full.

The two steps need different permissions, so run both with an **owner or admin**
token. Step 1 reads the dispatch queue, which requires `secret.read` — an auditor
token is refused with a 403 and never reaches step 2. Step 2 requires only
`run.read`, which every role carries.

Anything your `filter` writes with `console.log` appears there, alongside the
per-candidate verdicts the round recorded.

## See also

- [Architecture — global workflows](https://docs.kici.dev/architecture/global-workflows/) — dual-query dispatch flow, cross-provider auth, security model, lock-file schema.
- [Universal-git provider](https://docs.kici.dev/user/providers/universal-git/#global-workflows) — how global workflows interact with `generic:<orgId>:<sourceId>` routing keys.
- [SDK reference](https://docs.kici.dev/user/sdk-reference/) — the full set of triggers that accept `repos:`.

---

## Idempotent steps and check mode

Source: https://docs.kici.dev/user/idempotent-steps/

An **idempotent step** describes _desired state_ rather than a fixed sequence of
commands. You give the step a `check` function that inspects the world and a
`run` function that converges it. KiCI then executes the workflow in one of two
modes:

- **Apply mode** (the default): for each step, `check()` runs first; on drift the
  step applies the change; when already in sync the step is skipped.
- **Check mode** (`--check`): for each step, `check()` runs and KiCI reports what
  _would_ change — **without changing anything**. This is the same model as a
  dry-run plan: you see the drift before any side effect happens.

This turns a workflow into convergent configuration management: re-running an
apply is safe (in-sync steps do nothing), and a check-mode run is a read-only
preview you can gate a build on.

## Authoring a checked step

Add a `check` facet to the existing `step()` factory. When `check` is present,
`run` becomes the _apply_ function and receives the drift value `check`
returned:

```typescript
import { step, z } from '@kici-dev/sdk';

const configureNginx = step('configure-nginx', {
  // optional schema for the drift value — gives the dashboard a typed shape
  drift: z.object({ want: z.string() }),

  // read-only inspection; return null when already in the desired state
  check: async (ctx) => {
    const current = await ctx.$`nginx -T`;
    return current.stdout.includes(DESIRED) ? null : { want: DESIRED };
  },

  // human-readable preview line — REQUIRED when check is set. It is the drift's
  // serializable face: it streams to the logs and persists for the dashboard.
  summarize: (drift) => `would rewrite nginx.conf (${drift.want.length} bytes)`,

  // apply — runs only when check returned drift (apply mode); receives that drift
  run: async (ctx, drift) => {
    await writeConfig(drift.want);
    return { reloaded: true };
  },

  // optional — runs when check returned null, to produce the step's outputs
  whenInSync: async () => ({ reloaded: false }),
});
```

### The facet fields

| Field        | Required         | Purpose                                                                |
| ------------ | ---------------- | ---------------------------------------------------------------------- |
| `check`      | to opt in        | Read-only inspection. Return a drift value, or `null` when in sync.    |
| `summarize`  | when `check` set | Human-readable, serializable preview of the drift. Streams + persists. |
| `run`        | always           | Apply function. With `check`, it receives the drift as its second arg. |
| `whenInSync` | optional         | Produces the step's outputs when `check` returned `null`.              |
| `drift`      | optional         | Schema that validates / shapes the drift value.                        |

`summarize` is **required** whenever `check` is declared. `run` and `whenInSync`
both produce the same output type — one output shape per step, whichever path
runs. Every other step facet (`cache`, `rules`, `continueOnError`, `timeout`,
`retry`, `approval`, `onCancel`, `cleanup`, `outputs`) composes unchanged.

A plain `step()` without `check` keeps its exact current behavior — the check
facet is fully optional.

## Run modes

A run carries one of three modes:

| Mode                  | CLI flags                 | Behavior                                                                                   |
| --------------------- | ------------------------- | ------------------------------------------------------------------------------------------ |
| `apply`               | (default, no flags)       | Converge: drift ⇒ apply ⇒ **applied**; null ⇒ **in sync** (skipped).                       |
| `check`               | `--check`                 | Preview only: drift ⇒ **would change**; null ⇒ **in sync**. Never applies. Always exits 0. |
| `check-fail-on-drift` | `--check --fail-on-drift` | Same as check, but the run **fails** if any step reports drift.                            |

Per-step outcomes:

- **applied** — drift was found and the step applied the change (apply mode).
- **in sync** — `check` returned `null`; nothing to do.
- **would change** — drift was found in check mode; the change was previewed, not applied.
- **no check** — a plain step (no `check`) reached under check mode. A
  side-effecting step can't be safely previewed, so it is skipped.

In check mode KiCI never invokes a checked step's `run` (apply) — the preview is
guaranteed side-effect-free.

## Running in check mode

`--check` and `--fail-on-drift` control drift reporting on `kici run remote`:

```bash
# Apply (default): converge the workflow.
kici run push --local
kici run remote my-fixture

# Check: report drift, change nothing. Always exits 0.
kici run remote my-fixture --check

# Check + fail on drift: fail the run when any step reports drift. Use this as a
# CI gate ("fail the build if prod has drifted").
kici run remote my-fixture --check --fail-on-drift
```

`--fail-on-drift` only modifies check mode — passing it without `--check` is an
error.

## Where outcomes show up

A check-mode run is labeled in the dashboard with a **CHECK MODE — preview**
badge on the run header. Each step shows its outcome chip — applied / in sync /
would change / no check — and, when drift was detected, the `summarize` line
describing what would change. The rendering is read-only.

## See also

- [Idempotent SDK helpers](https://docs.kici.dev/user/sdk/idempotent/) — the `idempotent()` / `idempotentStep()` convenience wrappers (always apply on drift), plus `checkStep()`, the clean-shape sibling that respects the run-level check mode.
- [Core SDK reference](https://docs.kici.dev/user/sdk/core/) — the `step()`, `job()`, and `workflow()` factories the check facet extends.
- [Lock file and drift](https://docs.kici.dev/user/lock-file-and-drift/) — how the lock file carries step capability flags.

---

## Autoscaling workflows

Source: https://docs.kici.dev/user/workflows/autoscaling-workflows/

The [event scaler backend](https://docs.kici.dev/operator/orchestrator/event-scaler/) turns cloud autoscaling into ordinary workflow authoring. When the orchestrator needs a new agent, the event scaler emits a `kici.scaler.scale-up` event. When an agent is no longer needed, it emits a `kici.scaler.scale-down` event. You write two workflows: one that boots a cloud instance on scale-up, and one that deletes it on scale-down.

No cloud SDK ships inside KiCI. Your workflow calls the cloud provider's API directly. The examples below target Hetzner Cloud, but the same shape fits any provider with a create/delete API.

This page assumes you know the [`kiciEvent()`](https://docs.kici.dev/user/sdk/triggers/) trigger and [custom events](https://docs.kici.dev/user/events/). For the full event payloads, see the [event contract reference](https://docs.kici.dev/operator/orchestrator/event-scaler-events/).

The SDK exports the two event names and their payload schemas, so you subscribe with the same constant the scaler emits and parse the payload instead of casting it. Import `SCALER_EVENT_NAMES`, `ScalerScaleUpPayload`, `ScalerScaleDownPayload` and `ScaleDownReason` from `@kici-dev/sdk` — see [validation and events](https://docs.kici.dev/user/sdk/validation-events/#event-scaler-events).

## The provisioning workflow

The provisioning workflow subscribes to `kici.scaler.scale-up` and matches on the scaler name. It reads the payload from `ctx.rawPayload`, forwards the single-use claim code into a cloud instance, and boots that instance. The agent claims its own token in-instance and registers with the given `agentId`.

```ts
import {
  workflow,
  job,
  kiciEvent,
  buildAgentCloudInit,
  SCALER_EVENT_NAMES,
  ScalerScaleUpPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';

export default workflow('hetzner-autoscale-provision', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleUp, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('provision', {
      runsOn: ['default'],
      // Bind the context that holds the credential this job reads. A job
      // resolves only the secrets of the contexts it binds.
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleUpPayload.parse(ctx.rawPayload);

        // Forward the single-use claim code into cloud-init. The agent claims
        // its own token in-instance, so the token never transits provisioning.
        const userData = buildAgentCloudInit(
          {
            claimCode: payload.claimCode,
            agentId: payload.agentId,
            orchestratorUrl: payload.orchestratorUrl,
            labels: payload.labels,
          },
          {
            maxLifetimeMinutes: 30,
            deliveryMode: 'container',
          },
        );

        const token = await ctx.secrets.get('HETZNER_API_TOKEN');
        const res = await fetch('https://api.hetzner.cloud/v1/servers', {
          method: 'POST',
          headers: {
            Authorization: `Bearer ${token}`,
            'Content-Type': 'application/json',
          },
          body: JSON.stringify({
            name: `kici-agent-${payload.agentId}`,
            server_type: 'cpx12',
            image: 'debian-12',
            user_data: userData,
            // Every teardown layer keys off these labels.
            labels: {
              'kici-managed': 'hetzner-autoscale',
              'kici-agent-id': payload.agentId,
              'kici-scaler': SCALER_NAME,
            },
          }),
        });
        if (!res.ok) throw new Error(`Hetzner create failed: ${res.status}`);
        ctx.log.info(`Provisioned instance for agent ${payload.agentId}`);
      },
    }),
  ],
});
```

Read the [event contract reference](https://docs.kici.dev/operator/orchestrator/event-scaler-events/) for every payload field. The `agentId` correlates the spawn, so the instance must register with exactly that id.

### The bound context gates the scale-up

A provisioning workflow is an [event-triggered run](https://docs.kici.dev/user/events/#what-an-event-triggered-run-resolves), so it obeys every protection rule on the [context](https://docs.kici.dev/user/contexts/#protection-rules) it binds. That is what makes the cloud credential resolve, and it also means an approval hold stops the scale-up.

Read the chain, because the symptom sits far from the cause:

1. The scaler emits `kici.scaler.scale-up` and reserves a claim for the new agent.
2. The provisioning workflow matches, and its context holds the run for approval.
3. No instance boots, so no agent registers against the claim.
4. The event-provision reaper reaps the stranded claim once it expires.
5. The queued jobs wait, and you see agents that never appear.

Nothing in that chain reports "waiting for an approval" at the fleet level. Check the [approval queue](https://docs.kici.dev/user/dashboard/contexts-and-secrets/#approval-queue): it lists the held run, names the context that holds it, and gives the reason. The run-detail page shows the same hold under its approval block, and `kici runs show <run-id>` prints it from the terminal.

A provisioning workflow also needs an agent labelled `kici:role:builder` to pack its source before its own job runs. Keep at least one builder-role agent outside the pool the scaler provisions, so a scale-up never waits on the fleet it is scaling.

A [branch restriction](https://docs.kici.dev/user/contexts/#branch-restrictions) rejects the run outright. The orchestrator mints a scaler event itself, with no run behind it, so a scale-up carries no branch for a pattern to match.

Bind provisioning and teardown workflows to a context that carries the cloud credential and **no** approval hold, no wait timer, no branch restriction, and no `minimumTrust` gate. Gate a human-triggered deploy workflow instead. Keep the credential in that ungated context narrow: the scaler's own token, scoped to create and delete instances, and nothing else.

## The cloud-init that starts the agent

`buildAgentCloudInit(creds, options)` renders the `#cloud-config` that boots the KiCI agent. In the claim-code form it writes the single-use claim code — never a token — into a root-only env file (`0600`, owned by root). The agent exchanges that code for its own token inside the instance, so the token never transits cloud-init, the instance metadata, or any other provisioning channel. The env file holds:

- `KICI_ORCHESTRATOR_URL` — from `creds.orchestratorUrl`.
- `KICI_SCALER_CLAIM_CODE` — from `creds.claimCode`. The agent exchanges it for its own token in-instance.
- `KICI_AGENT_ID` — from `creds.agentId`.
- `KICI_LABELS` — from `creds.labels`, comma-joined.
- `KICI_SCALER_MANAGED=1` — marks the agent as scaler-managed, so it self-drains on idle and shutdown.

The `0600` env file still protects the non-secret env from other users on the instance. The claim code it carries is single-use and short-lived, so even that value is spent the moment the agent claims its token.

`maxLifetimeMinutes` is the one required option. It adds a max-lifetime self-poweroff (teardown layer L2): an instance that never receives a scale-down still removes itself after a hard cap. `deliveryMode` selects how the agent binary arrives — `'container'` runs the published agent image, `'payload'` fetches it from the orchestrator.

### Customization axes

Pass any of these options to shape the boot:

- `packages` — extra apt/yum packages, merged into the cloud-init `packages:` list.
- `writeFiles` — extra `write_files` entries (path, content, permissions, owner). The reserved env-file path is rejected, so a custom file cannot overwrite the credentials.
- `runcmdBefore` / `runcmdAfter` — shell lines that run before or after the agent starts.
- `agentEnv` — extra variables appended to the agent env file. Keys must be valid env names, and a value with a newline is rejected.
- `baseCloudConfig` — a raw cloud-config document to merge everything into (users, ssh keys, apt mirrors, mounts, bootcmd). The builder unions its `packages`, `runcmd`, and `write_files` with yours.

## The teardown workflow

The teardown workflow subscribes to `kici.scaler.scale-down` and deletes the instance registered under the scaled-down `agentId`. It finds the instance by the `kici-agent-id` label that the provisioning workflow set.

```ts
import {
  workflow,
  job,
  kiciEvent,
  SCALER_EVENT_NAMES,
  ScalerScaleDownPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';

export default workflow('hetzner-autoscale-teardown', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleDown, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('teardown', {
      runsOn: ['default'],
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleDownPayload.parse(ctx.rawPayload);
        const token = await ctx.secrets.get('HETZNER_API_TOKEN');

        const list = await fetch(
          `https://api.hetzner.cloud/v1/servers?label_selector=kici-agent-id==${payload.agentId}`,
          { headers: { Authorization: `Bearer ${token}` } },
        );
        const { servers } = (await list.json()) as { servers: Array<{ id: number }> };
        if (servers.length === 0) {
          ctx.log.info(`No instance found for agent ${payload.agentId}; nothing to tear down`);
          return;
        }
        for (const server of servers) {
          await fetch(`https://api.hetzner.cloud/v1/servers/${server.id}`, {
            method: 'DELETE',
            headers: { Authorization: `Bearer ${token}` },
          });
          ctx.log.info(`Deleted instance ${server.id} for agent ${payload.agentId}`);
        }
      },
    }),
  ],
});
```

Keep the teardown idempotent. "None found" logs and succeeds, and a delete that returns "already gone" is not an error. A scale-down can arrive after the instance already removed itself through the self-poweroff backstop.

## Guaranteed teardown

The scale-down workflow is the primary teardown path, but it is not the only one. The reference Hetzner implementation guarantees teardown with five independent layers, keyed off the resource labels every instance carries. The host-side reaper is the backstop that survives a crash or reboot. See the [teardown reaper runbook](https://docs.kici.dev/operator/orchestrator/hetzner-autoscale-reaper/) for the full model and the recommended alert.

## Running on other clouds (AWS / GCP / Azure)

The architecture is cloud-agnostic. Nothing inside KiCI is provider-specific. Only two things change per cloud: the API calls in your provision and teardown workflows, and the host-side reaper (teardown layer L4). You still subscribe to the same `kici.scaler.scale-up` / `kici.scaler.scale-down` events, and the agent still self-claims from the forwarded claim code the same way on every cloud, through the same `buildAgentCloudInit` call.

The five teardown layers map to each cloud's own idiom:

| Layer                              | Hetzner                    | AWS                         | Azure                               | GCP                      |
| ---------------------------------- | -------------------------- | --------------------------- | ----------------------------------- | ------------------------ |
| **L1 — Scale-down workflow**       | same event-driven workflow | same                        | same                                | same                     |
| **L2 — In-instance self-poweroff** | same cloud-init            | same cloud-init             | same cloud-init                     | same cloud-init          |
| **L3 — Harness finalizer**         | same                       | same                        | same                                | same                     |
| **L4 — Out-of-band reaper**        | host systemd timer         | tag-scoped scheduled Lambda | auto-shutdown or scheduled Function | scheduled Cloud Function |
| **L5 — Pre-suite sweep**           | same                       | same                        | same                                | same                     |

Only L4 has a real per-cloud shape, because it runs outside your instances and outside KiCI. Everything else is identical across providers.

### AWS provision workflow shape

The provision workflow keeps the same structure. It swaps the cloud call for the AWS EC2 SDK, and encodes the cloud-init as base64 because AWS `UserData` expects base64 (Azure `customData` wants base64 too). Read the AWS credentials from `ctx.secrets` and tag every instance so each teardown layer can find it.

```ts
import {
  workflow,
  job,
  kiciEvent,
  buildAgentCloudInit,
  SCALER_EVENT_NAMES,
  ScalerScaleUpPayload,
} from '@kici-dev/sdk';
import { EC2Client, RunInstancesCommand, ResourceType } from '@aws-sdk/client-ec2';

const SCALER_NAME = 'aws';

export default workflow('aws-autoscale-provision', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleUp, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('provision', {
      runsOn: ['default'],
      context: 'aws-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleUpPayload.parse(ctx.rawPayload);

        // Forward the claim code; the agent self-claims its token in-instance.
        // AWS UserData expects base64. Azure customData does too.
        const userData = buildAgentCloudInit(
          {
            claimCode: payload.claimCode,
            agentId: payload.agentId,
            orchestratorUrl: payload.orchestratorUrl,
            labels: payload.labels,
          },
          {
            maxLifetimeMinutes: 30,
            deliveryMode: 'container',
            userDataEncoding: 'base64',
          },
        );

        const client = new EC2Client({
          region: 'us-east-1',
          credentials: {
            accessKeyId: await ctx.secrets.get('AWS_ACCESS_KEY_ID'),
            secretAccessKey: await ctx.secrets.get('AWS_SECRET_ACCESS_KEY'),
          },
        });

        await client.send(
          new RunInstancesCommand({
            ImageId: 'ami-00000000000000000', // customer-supplied AMI with docker
            InstanceType: 't3.micro',
            MinCount: 1,
            MaxCount: 1,
            UserData: userData,
            // Every teardown layer keys off these tags.
            TagSpecifications: [
              {
                ResourceType: ResourceType.instance,
                Tags: [
                  { Key: 'kici-managed', Value: 'aws-autoscale' },
                  { Key: 'kici-agent-id', Value: payload.agentId },
                  { Key: 'kici-scaler', Value: SCALER_NAME },
                ],
              },
            ],
          }),
        );
        ctx.log.info(`Provisioned EC2 instance for agent ${payload.agentId}`);
      },
    }),
  ],
});
```

The teardown workflow mirrors this: it runs `DescribeInstances` filtered by the `kici-agent-id` tag, then `TerminateInstances` on the matches. "None found" logs and succeeds.

The AWS reference lives at `e2e/fixtures/aws-autoscale/`. It is compiled and typechecked against the AWS EC2 SDK, but it is not run against real AWS — unlike the Hetzner reference, which has a real-cloud E2E. Adapt the AMI, instance type, subnet, and IAM instance profile for your account.

## GitHub Actions runners

A provisioning workflow does not have to boot a cloud VM. Instead of a create/delete API, it can dispatch a GitHub Actions run that boots a one-shot agent. The agent self-claims from the forwarded claim code, registers with the orchestrator, runs exactly one job, and exits.

The scaler entry names the repo that holds the provisioning and teardown workflows, exactly as any other event backend does:

```yaml
scalers:
  - name: github-actions
    type: event
    maxAgents: 20
    provisioningTargets:
      - myorg/infra
    labelSets:
      - labels: [github-actions]
```

On `kici.scaler.scale-up`, the provisioning workflow dispatches a `kici-agent.yml` workflow run in a GitHub repo. It passes the claim code, orchestrator URL, agent id, and labels as dispatch inputs. The token never appears in those inputs — only the single-use claim code, which the agent exchanges for its own token in-instance.

The `kici-agent.yml` run starts the agent on the runner itself with `KICI_SCALER_CLAIM_CODE` set. `KICI_SCALER_MANAGED=1` and a zero idle timeout make the agent register, run one job, and exit. The GitHub Actions run then completes on its own. By default the run installs the published agent from npm; set `agent_bundle_release` to a release tag holding a `kici-admin agent package` tarball to pin an exact build or to serve runners that cannot reach npm.

Teardown is largely automatic. A GitHub Actions run self-completes when its agent exits. So the `kici.scaler.scale-down` workflow only cancels a run GitHub has not yet marked finished, and only for reasons where the agent will never do useful work (`spawn-timeout`, `heartbeat-timeout`).

Every other reason leaves the run alone to reap itself. That includes the reason a healthy one-shot agent produces when it exits after its job — cancelling there would turn a succeeding run into a cancelled one.

Both workflows read a `GITHUB_DISPATCH_TOKEN` [scoped secret](https://docs.kici.dev/user/secrets/) with `actions: write` permission on the target repo — provisioning to dispatch a run, teardown to cancel one. Bind the context that holds it on the job (`context: 'github-actions'`) — a job reads only the secrets of the contexts it binds. The runner workflow is at [`examples/github-actions-autoscale/`](https://github.com/kici-dev/kici-public/tree/main/examples/github-actions-autoscale), ready to copy into your runner repo’s `.github/workflows/`.

Both workflows read `GITHUB_RUNNER_REPO` as an org-level context variable on the same `github-actions` context that holds the secret — `kici-admin variable set <orgId> github-actions GITHUB_RUNNER_REPO --value myorg/ci-runners`. The provisioning workflow reads one more, the optional `GITHUB_AGENT_BUNDLE_RELEASE`. Copy [`provision.workflow.ts`](https://github.com/kici-dev/kici-public/blob/main/examples/github-actions-autoscale/provision.workflow.ts) and [`teardown.workflow.ts`](https://github.com/kici-dev/kici-public/blob/main/examples/github-actions-autoscale/teardown.workflow.ts) into your `.kici/workflows/`.

---