Upgrade and rollback

Every KiCI package ships at a single shared version, so “upgrading KiCI” means moving the orchestrator and agents to the same new release. This page is the upgrade runbook for both deployment shapes — the container image and the native service install.

Before you upgrade

Read the release notes for the target version so you know what changed.
Snapshot your PostgreSQL database first, with kici-admin db backup or your standard tooling. A migration down() can lose data, so the snapshot is the only path back to an older version that keeps all of your data.

Safe order: orchestrator first, then agents

Upgrade the orchestrator first, then the agents. When a release raises the minimum accepted protocol version, move both tiers in the same maintenance window. Protocol 4 raises it from 3: an orchestrator on protocol 4 refuses an agent from 0.15.x or earlier (protocol 3) at connect until the agent is upgraded.

Ephemeral scaler-spawned agents run the image (or binaryPath) their label set names, not the orchestrator’s version. A container agent moves to the new release only when you set its label set’s image to the new quay.io/kici-dev/kici-agent:<version> and restart the orchestrator; each new agent then pulls the new image. A label set that names kici-agent:latest with the default imagePullPolicy: IfNotPresent keeps the image the host already pulled, so pin the version, or set imagePullPolicy: Always to pull latest on every spawn.
Long-lived installed agents are upgraded with kici-admin agent upgrade.

On version skew: the orchestrator enforces a protocol-version floor during the agent handshake and rejects any agent below it with close code 4005 (Unsupported protocol version). The agent also reports its application version, but that is display metadata only — nothing warns or rejects on an application-version mismatch. Running the orchestrator and agents on the same release is the supported configuration. Transient skew while a rolling upgrade is in flight is tolerated only between releases that share a protocol floor; across a floor change (protocol 3 → 4) every not-yet-upgraded agent stays disconnected until its own upgrade, so upgrade the agents right after the orchestrator.

Agents and locks share a compile schema. An agent runs only a lock compiled for the compile schema it implements. When a release changes that schema, pin the new agent image (or binaryPath) and recompile your locks with the same release’s @kici-dev/compiler together. Until a lock is recompiled, every run of it fails. See compile schema mismatch.

Upgrade the coordinators of one cluster together. A release can change how the orchestrator stores queued jobs, and an older coordinator cannot read what a newer one wrote. For example, a coordinator from before queued job secrets were encrypted dispatches the jobs a newer coordinator queued without their secrets. Upgrade every coordinator in the same maintenance window, and do not leave an older one running next to a newer one.

Upgrade every coordinator and worker of a cluster together. A release with mutual peer authentication changes how peers connect. Orchestrators with mutual peer authentication and older orchestrators do not connect to each other in either direction, so until the last one is upgraded the cluster splits. The upgraded side logs Peer does not support mutual authentication; upgrade every orchestrator in the cluster, and the older side’s connection is refused with Mutual peer authentication required. Single orchestrators, the hosted Platform and agents are not affected.

Join with a matching kici-admin. kici-admin join and the cluster’s orchestrators must both be on a release with join protocol v2; there is no mixed-version join. Upgrade the cluster’s orchestrators, then the kici-admin you join with. Join tokens created before the upgrade keep working until they expire.

Drain before upgrading

Restarting an orchestrator to upgrade it severs the WebSocket connection to any agent currently running a job. To avoid that, drain the orchestrator first: draining stops it from dispatching any new job while letting the jobs already running on agents finish. When no jobs are running the orchestrator is quiesced and safe to restart.

# Stop new dispatches and block until in-flight jobs finish, then upgrade.
kici-admin orchestrator drain --wait && \
  kici-admin orchestrator upgrade

kici-admin orchestrator drain flips the orchestrator into draining and prints draining=true jobsRunning=<n>. New webhooks still ingest and match — the resulting jobs are held as Pending in the durable queue rather than sent to an agent.
--wait polls until jobsRunning reaches 0 (quiesced), then exits 0. It exits 2 if the wait times out with jobs still running (default timeout 300s, tune with --timeout <seconds>), and 1 on error — so the && chain above only proceeds to the upgrade once the orchestrator is genuinely idle.
A run that is mid-flight pauses across the upgrade: its next jobs stay Pending until the fresh orchestrator comes back up, which automatically re-dispatches the held backlog through its normal startup recovery. No work is lost.
Draining is in-memory only. A restart (the upgrade) comes back accepting new jobs and immediately drains the Pending backlog — there is no drain flag to clear afterwards.
To abort a drain without restarting (resume dispatching), run kici-admin orchestrator resume. Check the current state any time with kici-admin orchestrator drain --status.

Draining requires the same orchestrator-admin authentication as the other kici-admin orchestrator-plane commands (an admin API key of the owner or admin role).

Database migrations run at startup

On boot the orchestrator applies any pending schema migrations automatically. Disable this with KICI_AUTO_MIGRATE=false if you prefer to run migrations out of band. In a multi-orchestrator cluster a PostgreSQL advisory lock ensures exactly one instance migrates while the others wait, so a rolling restart never races the schema.

With KICI_AUTO_MIGRATE=false, migrate before you start the new binary. The orchestrator does not check the schema at startup. An un-migrated database produces no boot error, only per-request write failures for whatever the new release added. Run kici-admin db migrate --database-url <url> from the new release first. This form migrates the database directly, so the orchestrator does not need to run. kici-admin db migrate without --database-url asks a running orchestrator to migrate over its admin API.

Inspect applied migrations with kici-admin db migrate --status. Detect migration drift with kici-admin db check-schema. Compare the live schema with the schema the release expects with kici-admin db schema-diff.

Upgrading each deployment shape

Container / compose

Bump the pinned quay.io/kici-dev/kici-orchestrator tag or digest and redeploy. Published version tags are immutable, so a given tag never changes underneath you. Do the same for the agent image referenced in the scaler config.

Native service install

Use kici-admin orchestrator upgrade and kici-admin agent upgrade. Both use a versioned directory layout with an atomic symlink flip, and previous versions are preserved on disk. See Service installation for the full command reference.

A native install from npm runs on the Node.js of the host. Node.js checks TLS certificates against its own built-in certificate store, and only a Node.js update changes that store. Keep the host on the newest Node.js 24 patch release. A full package carries its own Node.js, and an upgrade to a newer release replaces it. A light package runs on the Node.js version that its release names: before the upgrade, install that version into the cache.

To move a native install from npm to a new Node.js runtime, such as Node.js 24.22.1:

Install the new Node.js runtime on the host, at a fixed path. Do not use a version-manager shim.
Move the orchestrator: kici-admin orchestrator upgrade --version <version> --node <path-to-node>. The command installs the package with the npm of that runtime, registers the service again to run it, and restarts it. A scheduled db backup unit moves with it.
Move each agent the same way: kici-admin agent upgrade --version <version> --node <path-to-node>.
If a bare-metal scaler starts agents, set its binaryPath to the kici-agent of the new runtime. The upgrade prints a warning that names this path, and the “Agent binary” line of kici-admin orchestrator status shows it. Then restart the orchestrator.

See --node to move to another Node.js runtime for the checks the upgrade does first.

Before it restarts the orchestrator, kici-admin orchestrator upgrade reads the scaler config (KICI_SCALER_CONFIG_PATH and KICI_SCALER_CONFIG_DIR) and warns about each container label set that would keep spawning an old agent:

a kici-agent:<version> older than the upgrade target
kici-agent:latest (or no tag) with imagePullPolicy absent or IfNotPresent

Each warning names the scaler, the label set and the image: line to set. The upgrade still runs: edit the scaler config, then restart the orchestrator (kici-admin orchestrator restart) so new agents use the new image.

After the restart, kici-admin orchestrator upgrade uploads the fleet agent payloads for the new version to the cache bucket. It reads the cache storage config (KICI_STORAGE_*) and the S3 credentials (AWS_*) from the env file of the instance that it upgrades. This is the env file that the service runs with. The KICI_STORAGE_* variables of your shell are not used, so you do not have to export the storage config before you upgrade. See Auto-package on orchestrator upgrade for the full rules.

Once the upgraded service runs, kici-admin orchestrator upgrade reports the compile schema of the registered workflow locks through the admin API. It prints one warning per lock that the new release’s agents reject, or All registered workflow locks match compile schema <N>. Without an admin token, or when the API does not answer, it prints a line that points to kici-admin diagnose instead. The report never fails the upgrade.

Rolling back

Native service install — kici-admin orchestrator upgrade --rollback returns to the previous installed version (it requires at least two installed versions). --pick selects any installed version interactively, and --cleanup keeps the current and previous versions while pruning older ones.
Container / compose — re-pin the previous immutable image tag or digest and redeploy. There is no rollback subcommand for image deployments. The 0.8.0 retirement removed every earlier tag and its manifest from quay.io, so a version below 0.8.0 does not pull. Such an image is available only from a copy you still hold: your local image store or your own registry mirror.

Roll the agents back before the orchestrator. Newer agents self-report kici:agent-feature: labels, such as kici:agent-feature:global-eval-skips-result-aware. An orchestrator older than those agents does not know the prefix, so it treats the label as one the agent’s token must allow. An agent token scoped to a label set then refuses the agent’s registration. If you must roll back the orchestrator alone, register the newer agents with unscoped tokens until you roll them back too.

Rolling the software back does not roll the database schema back. An older orchestrator running against a newer schema is unsupported. Revert the schema with kici-admin db migrate --to <migration> while the newer version still runs, because only that version has the down() functions of its own migrations. See Service installation for the native-service rollback. Or restore the database snapshot you took before the upgrade.

Software rollback is not config rollback

kici-admin config rollback --to N rewinds the shared orchestrator configuration history and is unrelated to the software version. See Config management for that workflow.