Service installation guide

This guide covers installing and managing the KiCI orchestrator and agent as native system services using the kici-admin CLI.

Pre-release note: This version introduces folder-anchored instance targeting. The on-disk layout is breaking — per-instance config, log, and install directories are now name-scoped under <root>/<name>/. Re-install any existing instance to adopt the new layout; there is no automatic migration.

Overview

The kici-admin CLI provides commands to install, manage, and upgrade the orchestrator and agent as native services on:

• Linux — systemd (system-level or user-level)
• macOS — launchd (system-level or user-level)
• Windows — Windows Services via shawl
• Docker/Podman — Compose file generation

Commands are organized into two groups:

kici-admin orchestrator install|uninstall|start|stop|restart|status|logs|upgrade
kici-admin agent install|uninstall|start|stop|restart|status|logs|upgrade

Instance directory and manifest

Each kici-admin orchestrator install (and kici-admin agent install) writes a manifest into a deploy folder — .kici-orchestrator.json for the orchestrator, .kici-agent.json for the agent. The deploy folder is the instance directory: pass --instance-dir <path> to choose it explicitly; the default is the current working directory. Every subsequent kici-admin <component> <cmd> resolves its target via this manifest — either by running the command from inside the deploy folder, or by passing --instance-dir <path> explicitly.

Re-running install for an instance whose deploy folder already holds its manifest updates that manifest. It does not replace it. The migration heads the instance recorded (see The schema precondition) and the time of the first install stay. The paths and the platform come from the new install. The recorded version is the version of the launcher the service definition starts: the version of its npm package, or the version folder under the install base that the launcher is in. When the launcher does not show a version, for example a --binary outside the install base, the version recorded before stays. A compose install records the release of the image its compose file pins. When the compose file pins :latest, the version recorded before stays. When the manifest cannot be read, or belongs to an instance with another name, install replaces it and prints a warning. To keep the env file, pass --no-wizard or the same --env-file. On a terminal, install otherwise runs the setup wizard, which writes a new env file.

The host’s installed instances are tracked at an instance index:

• User-level: ~/.config/kici/instances.json (Linux), ~/Library/Application Support/kici/instances.json (macOS), %LOCALAPPDATA%\kici\instances.json (Windows).
• System-level: /etc/kici/instances.json (Linux/macOS), C:\ProgramData\kici\instances.json (Windows).

The index is a reconciled cache, not the source of truth. install also embeds the deploy folder directly in the generated service definition (an X-KiCI-InstanceDir directive in the systemd unit, the equivalent plist key / container label / service-description marker on other platforms). Every command rebuilds the index from the init system’s native scan: it drops stale entries whose units are gone, and it re-adopts the deploy folder straight from the unit marker when the index entry is missing. So even if the index file is deleted or emptied, --name resolution still finds the deploy folder and the index self-heals on the next command. (Units installed before the marker existed have no embedded folder; for those, re-run install once with --instance-dir to regenerate the service definition with the marker. On Unix, upgrade only swaps the version symlink and leaves the existing definition in place.)

To point the instance index (and the per-instance config dirs) at a directory other than the platform default, set KICI_CONFIG_ROOT. This isolates one host’s instances from another tool’s — useful when test harnesses install throwaway instances alongside a real one and must not share its index.

Name-scoped on-disk layout

Per-instance paths embed the service name as a directory segment, so two instances with different --name values are fully isolated:

• Config: <configRoot>/<name>/ (e.g. /etc/kici/<name>/, ~/.config/kici/<name>/).
• Logs: <logRoot>/<name>/ (e.g. /var/log/kici/<name>/, ~/.local/share/kici/<name>/logs/).
• Install base: <installBase>/<name>/ (e.g. /opt/kici/<name>/ on systemd, /usr/local/kici/<name>/ on launchd, C:\Program Files\KiCI\<name>\ on Windows).

Both instances can run different versions concurrently — upgrade flips the symlink only inside the resolved instance’s own <installBase>/<name>/ tree.

Multi-instance hosts

When running multiple orchestrators or agents on one host (for example, a production instance alongside a test instance), give each a distinct --name and a distinct --instance-dir. The install command guards against accidental collisions: it refuses to register a same-named instance at a different --instance-dir unless you pass --force. The error names the existing instance directory so you can choose between picking a different name, picking a different deploy folder, or explicitly overwriting.

Targeting operating commands

Every lifecycle command — uninstall, upgrade, start, stop, restart, status, logs — requires explicit targeting. The resolution priority is:

1. --instance-dir <path> flag.
2. --name <name> flag (matches against the reconciled instance list for the current privilege scope).
3. A manifest in the current working directory (./.kici-orchestrator.json or ./.kici-agent.json).

If none of those resolve, the command exits non-zero with a candidate list of installed instances on the host. --name no longer has a default — a bare kici-admin orchestrator uninstall outside any deploy folder will refuse and list the candidates.

Once the target resolves, the command reads the service platform from that install’s manifest. A Compose install is managed with Compose even on a systemd host, and a systemd install is managed with systemd even where a container runtime is present. --platform <type> overrides the manifest and forces one driver. Only install detects the platform from the host, because it is the command that writes the manifest.

Instance discovery is wider than the operation: --name and the candidate list scan every driver the host can run, so a Compose instance is listed and resolvable on a systemd host.

A driver that cannot read its own registry is skipped instead of reporting nothing, and the command says so on stderr, naming the platform and the reason. If the container daemon is down or hung, Compose instances do not appear in the candidate list. Their index entries are left untouched: an unreadable registry is not evidence that the instances are gone. This applies to every platform — an unreadable systemd unit directory, launchd job or Windows service registry is skipped on the same terms. Fix what the warning names and re-run the command to see those instances again.

stop and uninstall are idempotent on a not-found --name: if you pass --name <n> and no instance by that name is installed on the host, the command prints … is not installed — nothing to stop/uninstall. and exits 0 (there is nothing to remove). This is distinct from the ambiguous-targeting refusal above: supplying an explicit --name that isn’t present is a successful no-op. Cleanup loops that stop/uninstall a set of candidate names across a fleet therefore don’t fail on the hosts where a given name was never installed.

Prerequisites

You need one of:

• A full package (standalone, includes the Node.js binary and npm) — download it from the release, see Download a package
• Node.js 24.5.0 or later with the orchestrator package installed via npm

A kici-admin from a standalone package carries no orchestrator or agent server. With it, pass install --binary with the path of the launcher in the orchestrator or agent package, such as <package dir>/kici-orchestrator-standalone. Without --binary, install stops and asks for it.

For Windows, shawl is downloaded automatically on first install (cached for future use).

Quick start

The examples below pin each install to an explicit --instance-dir so subsequent lifecycle commands can resolve the manifest unambiguously. Omit --instance-dir to anchor the manifest in the current working directory; either way, run subsequent commands from the deploy folder or pass --instance-dir again.

Linux (systemd)

# System-level service (run as root)
sudo kici-admin orchestrator install --wizard --instance-dir /opt/kici-deploy

# User-level service (no sudo needed)
kici-admin orchestrator install --wizard --instance-dir ~/kici-deploy

# Start the service (resolves the manifest in CWD)
cd ~/kici-deploy
kici-admin orchestrator start

# Or target it explicitly from anywhere
kici-admin orchestrator status --instance-dir ~/kici-deploy

sudo with a version manager. When node comes from a version manager such as nvm or mise, sudo kici-admin … fails with command not found: sudo resets PATH to its own secure_path, which does not include the version manager’s install directory. Run the command without sudo and pass --system. The CLI refuses and prints the same command with the full paths of the node binary and the CLI script. Run that printed command.

macOS (launchd)

# User-level service (recommended)
kici-admin orchestrator install --wizard --instance-dir ~/kici-deploy

# System-level service
sudo kici-admin orchestrator install --wizard --instance-dir /opt/kici-deploy

# Start the service
kici-admin orchestrator start --instance-dir ~/kici-deploy

Windows

Open PowerShell as Administrator:

# Install with interactive wizard
kici-admin orchestrator install --wizard --instance-dir C:\kici-deploy

# Start the service
kici-admin orchestrator start --instance-dir C:\kici-deploy

# Check status
kici-admin orchestrator status --instance-dir C:\kici-deploy

The Windows service receives the path of its env file in KICI_ENV_FILE, never the values in the file, because any local account can read the command line of a service with sc.exe qc. The orchestrator or agent reads the env file when it starts. Thus an edit to the file takes effect at the next restart. Each line is KEY=value, and the value is everything after the first =, as written: the service removes no quotes, and a # in a value is part of the value. PATH lines in the file go in front of the PATH of the service. Node.js reads some variables, such as NODE_OPTIONS and NODE_EXTRA_CA_CERTS, only when it starts. When the file sets one of them, the process starts again with the loaded environment, so the variable applies as it does on Linux and macOS. A batch file that you give as --binary runs before the process reads the env file, so the batch file does not see the variables in it.

A service that an older CLI installed has the values of the env file on its command line. To register it again, upgrade it, or re-run install --no-wizard with the --instance-dir, --name and --binary of the first install, then start it. --no-wizard keeps the env file. Every upgrade on Windows registers the service again, with the registration of the kici-admin that runs the upgrade: an archive upgrade, --rollback, --pick, and an npm-source upgrade, --restart-only included. After an upgrade that an older kici-admin ran, register the service again with the new kici-admin. For an install from npm, run kici-admin <component> upgrade --restart-only --yes. For an install from a package, re-run install --no-wizard as above, because --restart-only cannot read the version of a package launcher.

A release from before KICI_ENV_FILE can run only with the values of the env file on its command line. So on Windows, upgrade refuses such a release: an archive upgrade, --rollback, --pick and an npm-source upgrade all refuse it before they change anything, and the service keeps running. install also refuses such a release as --binary. To run such a release anyway, uninstall the service, then install the release with the kici-admin of that release.

Docker/Podman Compose

# Generate a compose file
kici-admin orchestrator install \
  --platform compose \
  --instance-dir ./kici-orchestrator \
  --env-file ./orchestrator.env

# Start with Docker Compose
cd ./kici-orchestrator
docker compose up -d

Image pinning by manifest-list digest

The generated compose file pins the image by manifest-list digest, not the mutable :latest tag:

image: quay.io/kici-dev/kici-orchestrator:0.1.15@sha256:<index-digest>

The image is kici-orchestrator for an orchestrator and kici-agent for an agent, whatever --name the service has.

The digest is the multi-arch image-index digest recorded at release time, so the same reference resolves the correct image on both linux/amd64 and linux/arm64, and a docker/podman pull verifies the hash — a registry that serves substituted content fails the pull. The exact digests for each release are published on the release artifacts page.

Because the pin is baked into the generated file, upgrading means regenerating the installer from a newer kici-admin (re-run kici-admin orchestrator install from the version you want) — that writes a new compose file with the new release’s digest. Pulling a different image without regenerating is a deliberate manual edit of the pin.

If you generate the compose file from a development build that never ran a release, the installer falls back to the :latest tag (with a warning) since no recorded digest ships with that build.

Shutdown grace period

The generated compose file sets stop_grace_period so the runtime waits for the service to shut down cleanly:

stop_grace_period: 45s

Compose waits only 10 seconds by default. The orchestrator needs longer, because it stops each agent it manages before it exits. If the runtime stops waiting first, it kills the orchestrator part-way through shutdown. The orchestrator then never tells its peers that it is leaving, and never closes its agent connections. The generated value gives each component more time than its own shutdown budget, so the service always finishes on its own terms.

Installation modes

Wizard mode (default on an interactive terminal)

Interactive guided setup. On an interactive terminal a bare kici-admin orchestrator install runs the wizard by default; pass --wizard to force it, or --no-wizard to skip it and write a stub env file to edit by hand. The wizard asks only essential questions with sensible defaults:

1. Operating mode — Hybrid (the default: hosted Platform relay plus this orchestrator’s own direct webhook ingress), Platform (relay only), Observed, or Independent
2. Database URL — PostgreSQL connection string
3. Port — HTTP server port (default: 4000)
4. Secrets key — 32-byte hex encryption key (auto-generated with option to customize)
5. Platform connection (if Hybrid/Platform/Observed mode) — Platform URL and API key. Observed mode then asks for the orchestrator’s own public webhook base URL (required); Hybrid asks for the same URL as an optional answer — leave it blank to run relay-only for now and set KICI_WEBHOOK_PUBLIC_URL when you expose the ingress
6. GitHub App source (optional) — source name, App ID, private-key path, and webhook secret

kici-admin orchestrator install

If you opt into a GitHub App source, the install prints a ready-to-run kici-admin source add github ... command in its “Next steps”. Sources are added at runtime against a running orchestrator, so run that command once the orchestrator is started — it is not persisted into the env file.

In a non-interactive shell (CI, a provisioning script) the wizard is skipped automatically and install writes the stub env file, so scripted installs never block on a prompt.

Both paths write KICI_MODE=hybrid. Hybrid accepts every Platform-relayed webhook exactly as platform does and additionally serves the orchestrator’s own ingress route, which verifies the webhook signature before it acts on a delivery. It requires no configuration beyond what platform requires, so an orchestrator nothing posts to directly behaves the same as a relay-only one. Pass --mode platform to opt out.

Flags mode

Provide configuration via CLI flags or an existing env file. Passing --env-file, --no-wizard, or --dev also skips the wizard:

kici-admin orchestrator install \
  --env-file /path/to/orchestrator.env \
  --binary /usr/local/bin/kici-orchestrator \
  --name kici-orchestrator \
  --instance-dir ~/kici-deploy \
  --platform systemd

Available flags:

Flag	Description	Default
--platform <type>	Force platform: systemd, launchd, windows, compose	Detected from the host
--mode <mode>	Operating mode written to the env file: hybrid, platform, observed, independent	hybrid
--env-file <path>	Path to existing env/config file	Creates new in config dir
--binary <path>	Path to orchestrator binary	Current executable
--name <name>	Service name (also the per-instance directory segment under config/log/install roots)	kici-orchestrator
--instance-dir <path>	Deploy folder where the instance manifest is written and resolved from	Current working directory
--force	Overwrite a same-named instance already installed at a different --instance-dir	Off
--dev	Dev mode with local PostgreSQL	Off
--wizard	Run the interactive setup wizard	On (interactive terminal)
--no-wizard	Skip the wizard and write a stub env file to edit by hand	Off

Dev mode (--dev)

Sets up a local development instance with a PostgreSQL container:

kici-admin orchestrator install --dev

This will:

1. Detect Podman or Docker
2. Start a PostgreSQL 18 container on port 15432 (avoids conflicts with existing Postgres)
3. Create an env file with the container’s KICI_DATABASE_URL
4. Register the service

The container is named {service-name}-dev-pg and can be managed with your container runtime.

Agent installation

Agent installation follows the same folder-anchored pattern with different configuration. The default --name is kici-agent, and the manifest filename is .kici-agent.json.

# Wizard mode
kici-admin agent install --wizard --instance-dir ~/kici-agent-deploy

# Flags mode
kici-admin agent install \
  --instance-dir ~/kici-agent-deploy \
  --name kici-agent \
  --orchestrator-url http://orchestrator:4000 \
  --token <agent-token> \
  --labels "linux,docker"

Agent-specific flags:

Flag	Description
--orchestrator-url <url>	URL of the orchestrator to connect to
--token <token>	Agent authentication token
--labels <labels>	Comma-separated routing labels, for example linux,docker. Every agent also reports its kici:* labels
--port <port>	HTTP port for the agent’s /health, /ready and /metrics endpoints (default 8080)

The flags are written to the agent’s env file as KICI_ORCHESTRATOR_URL, KICI_AGENT_TOKEN, KICI_LABELS and KICI_PORT.

The same --instance-dir / --name / --force flags described above apply to agent install. Multiple agents on one host need distinct --name, --instance-dir and --port values: two agents on the same port cannot both start. The create-path guard refuses to clobber a same-named foreign agent unless --force is set.

File locations

All per-instance paths are name-scoped under the listed roots: replace {name} with the value of --name (defaults to kici-orchestrator or kici-agent).

Linux

Level	Config root + instance dir	Log root + instance dir	Systemd unit
System (root)	/etc/kici/{name}/	/var/log/kici/{name}/	/etc/systemd/system/{name}.service
User	~/.config/kici/{name}/	~/.local/share/kici/{name}/logs/	~/.config/systemd/user/{name}.service

Instance index: /etc/kici/instances.json (system) or ~/.config/kici/instances.json (user).

macOS

Level	Config root + instance dir	Log root + instance dir	Plist
System (root)	/etc/kici/{name}/	/var/log/kici/{name}/	/Library/LaunchDaemons/dev.kici.{name}.plist
User	~/Library/Application Support/kici/{name}/	~/Library/Logs/kici/{name}/	~/Library/LaunchAgents/dev.kici.{name}.plist

Instance index: /etc/kici/instances.json (system) or ~/Library/Application Support/kici/instances.json (user).

A KiCI plist file named com.kici.{name}.plist or {name}.plist is also instance {name}. The lifecycle commands manage it under the label that the plist declares. install replaces it with dev.kici.{name}: it unloads the old job and removes the old plist.

Windows

Level	Config root + instance dir	Log root + instance dir
System	C:\ProgramData\kici\{name}\	C:\ProgramData\kici\{name}\logs\
User	%LOCALAPPDATA%\kici\{name}\	%LOCALAPPDATA%\kici\{name}\logs\

Instance index: C:\ProgramData\kici\instances.json (system) or %LOCALAPPDATA%\kici\instances.json (user).

For a system-level service, install gives the config folder of the instance an ACL of its own. LocalSystem, the account the service runs as, and Administrators get full control, and other accounts get no access. Everything in the folder gets the same ACL, including the env file and the service logs. Read or edit them from an elevated shell. A user-level config folder is in the profile of its user, which other accounts cannot read, and install leaves its ACL unchanged.

Install base (versioned upgrade tree)

The upgrade command extracts each version into a name-scoped directory and flips a symlink:

Platform	Install base
systemd, compose	/opt/kici/{name}/
launchd	/usr/local/kici/{name}/
Windows	C:\Program Files\KiCI\{name}\

Two instances with different --name values therefore have independent versioned trees — upgrading one does not touch the other.

Service lifecycle commands

Every lifecycle command resolves its target through the same priority chain: --instance-dir > --name > a manifest in the current working directory. Without any of those, the command refuses and prints the candidate list of installed instances on the host. The examples below use --instance-dir; they work identically if you cd into the deploy folder first and drop the flag. The service platform comes from the resolved install’s manifest — see Targeting operating commands.

Start

kici-admin orchestrator start [--instance-dir <path>] [--name <name>] [--platform <type>]

On systemd, launchd and Windows, start does not restart a service that is already running.

Stop

kici-admin orchestrator stop [--instance-dir <path>] [--name <name>] [--platform <type>]

stop returns when the service has stopped, and the service stays stopped until you start it. stop also succeeds on a service that is already stopped, so you can run it more than once.

On macOS, launchd restarts a job whose process fails. So stop unloads the launchd job, and start loads it again from its plist. The plist stays installed, so launchd still starts the service at the next boot (system-level) or login (user-level). launchd gives the process 45 seconds (orchestrator) or 20 seconds (agent) to shut down before it kills the process. A service installed by an older CLI uses the launchd default of 5 seconds. To apply the longer period, re-run install for that service.

On Windows, stop sends the process a ctrl-C. The service wrapper then waits up to 45 seconds (orchestrator) or 20 seconds (agent) for the process to exit. Both processes finish their own shutdown within that time. When --binary is a batch file (.cmd or .bat), the service runs it through cmd.exe with its input from NUL. Thus the service stops when the process exits: cmd.exe does not wait at its Terminate batch job (Y/N)? prompt. Because its input is NUL, a command in the batch file that waits for input, such as timeout or pause, does not wait. install refuses a batch-file path that holds ", % or ^. When the path holds no space, install also refuses &, |, <, >, (, ), ,, ; and =. For a service installed by an older CLI, the wrapper kills what it runs 3 seconds after the ctrl-C. To apply the longer period, re-run install for that service, or upgrade it: every upgrade registers a Windows service again.

Restart

kici-admin orchestrator restart [--instance-dir <path>] [--name <name>] [--platform <type>]

Status

Shows OS-level service state (running/stopped/failed, PID, uptime) and names the config files this install owns. For a running service, it also reads the orchestrator’s /health and /ready endpoints. It finds them from the env file: the port in KICI_PORT (default 4000), the path prefix in KICI_BASE_PATH, and the address in KICI_HOST when that names one address rather than every interface:

kici-admin orchestrator status [--instance-dir <path>] [--name <name>] [--json]

Example output, for an install whose env file names a scaler config. An install that names none prints the Env file: line alone:

Service: kici-orchestrator
State:   running
PID:     12345
Uptime:  2h 15m

--- KiCI orchestrator ---
Health:     ok
Ready:      yes
Version:    0.11.0 (built 2026-09-25T15:27:17.455Z)
SDK:        0.11.0 (bundle b012bd8bace3)
Uptime:     2h 15m

--- Config files ---
Env file:      /etc/kici/kici-orchestrator/kici-orchestrator.env
Scaler config: /etc/kici/scalers.yaml
Agent binary:  /usr/local/bin/kici-agent

• Health is ok whenever the orchestrator answers /health.
• Ready is yes when every readiness check passes: database (the database answers) and warm (the orchestrator has finished starting). Otherwise it reads no and names the failing checks, for example no (failing: database). It reads unknown when /ready does not answer. The command waits for /ready a few seconds longer than the database pool waits for a connection (KICI_DB_POOL_ACQUIRE_TIMEOUT_MS), so a database that does not answer shows as a failing check.
• Version is the KiCI release the orchestrator was built from, with its build date. Version and SDK are the build fingerprint. The SDK bundle hash matches the one kici-admin agent status prints when the orchestrator and the agent were built from the same SDK.
• Uptime in this section is the orchestrator process’s own uptime.

A running service that does not answer on its port prints (Could not reach health API) in place of the section.

The Config files block comes from the install manifest and the env file on disk, so it prints for a stopped service too. A compose install also lists the generated compose file. The scaler line appears only when the env file names a scaler config.

The Agent binary line names the kici-agent that npm install -g kici-admin installed with the Node.js runtime the service runs. The command reads that runtime from the service unit. Use this path as the binaryPath of a bare-metal scaler: kici-admin orchestrator upgrade installs the new agent at this path, so the agent upgrades with the orchestrator. The line shows (not installed) when the file is missing. It does not appear for a compose install, or for a service that does not run from an npm install of kici-admin. The --json output carries the same value as agentBinary.

Use --json for machine-readable output. The health and readiness objects carry the /health and /ready responses, and the config paths come back as a configPaths object.

kici-admin agent status reports an agent the same way. It reads the agent’s /health endpoint on the port the agent’s env file sets in KICI_PORT (default 8080):

--- KiCI agent ---
Agent ID:     kici-agent-7f3a
Orchestrator: connected
Active jobs:  1
Version:      0.11.0
SDK:          0.11.0 (bundle b012bd8bace3)
Uptime:       42m 7s

Orchestrator is connected while the agent’s connection to its orchestrator is open. With --json, the health object carries the agent’s /health response.

Logs

Tail and follow service logs with filtering:

# Follow logs (default)
kici-admin orchestrator logs

# Recent logs without follow
kici-admin orchestrator logs --no-follow --since 30m

# Filter by level
kici-admin orchestrator logs --level error

# JSON output
kici-admin orchestrator logs --json --since 1h

Flag	Description
--since <duration>	Show logs since duration (e.g., 1h, 30m, 2d)
--level <level>	Filter: error, warn, or info
--json	Output as JSON lines
--no-follow	Don’t tail — print and exit

Platform-specific log sources:

• Linux: journalctl -u {name} (system) or journalctl --user-unit {name} (user)
• macOS: log files in the logs directory
• Windows: Windows Event Log (wevtutil)

Upgrade procedure

The upgrade uses a name-scoped versioned directory layout: each version is extracted under the resolved instance’s own install base (e.g. /opt/kici/<name>/orchestrator-0.3.0/), and a per-instance symlink (e.g. /opt/kici/<name>/orchestrator) points to the active version. This enables instant rollback and keeps previous versions available — and because the install base is name-scoped, upgrading one instance never touches another.

upgrade accepts --instance-dir and --name on the same priority chain as the other lifecycle commands. Run it from inside the deploy folder, or pass --instance-dir explicitly:

# Upgrade from a local archive (target resolved from CWD manifest)
cd ~/kici-deploy
kici-admin orchestrator upgrade --from /path/to/kici-orchestrator-standalone-0.3.0-linux-x64.tar.gz --version 0.3.0

# Or pass --instance-dir from anywhere
kici-admin orchestrator upgrade \
  --instance-dir ~/kici-deploy \
  --url https://github.com/kici-dev/kici-public/releases/download/v0.3.0/kici-orchestrator-standalone-0.3.0-linux-x64.tar.gz \
  --version 0.3.0

An archive upgrade of the orchestrator starts the kici-orchestrator-standalone launcher of the archive, which is the independent-mode orchestrator. Use it only for an orchestrator in independent mode (KICI_MODE=independent), with a kici-orchestrator-standalone package. --url downloads the archive without a checksum check. To verify it first, download it and SHA256SUMS as Download a package shows, and pass the verified file with --from.

The upgrade command:

1. Resolves the target instance (refuses if no target can be resolved).
2. Extracts the new version under the resolved instance’s <installBase>/<name>/ tree (e.g., /opt/kici/<name>/orchestrator-0.3.0/). On Windows, it first refuses a version from before KICI_ENV_FILE (see Windows), and extracts nothing.
3. Stops the running service.
4. Updates the per-instance symlink atomically (Unix) or re-registers the service (Windows).
5. Starts the service with the new version.

Previous versions of the resolved instance are preserved on disk for rollback. Other installed instances are untouched.

npm-source upgrade (no archive)

For deployments installed from npm, you can upgrade without supplying an archive. upgrade with no --from/--url is self-driving — a single command installs the correct global package under the unit’s own runtime, restarts, and verifies:

# Upgrade an npm-installed orchestrator to a published version.
kici-admin orchestrator upgrade --version <version> --yes

# The agent flavor works the same way
kici-admin agent upgrade --version <version> --yes

With no archive source, the self-driving upgrade:

1. Reads the launch command the installed service unit will actually execute (the ExecStart of the systemd unit, the ProgramArguments of the launchd plist, or the service binary path on Windows), and from it recovers the unit’s pinned node runtime and the global package its launch target resolves — kici-admin when the component is loaded through kici-admin’s nested node_modules, or the standalone @kici-dev/<component>.
2. Runs npm install -g <package>@<version> --prefix=<prefix>, where <prefix> is the npm global prefix that holds the unit’s launch target. The pinned node runs the npm-cli.js that ships beside it, so a version-manager wrapper at <pinned>/bin/npm does not run. The install lands in the global prefix the unit’s launch path resolves from, whichever node your shell has active and whichever prefix your environment names. For example, npx kici-admin sets npm_config_prefix to its own prefix, and the upgrade ignores it. Other npm settings in your environment, such as the registry, still apply. On Windows, it first asks the registry whether <version> reads KICI_ENV_FILE with npm view, and refuses a release from before it (see Windows) before npm installs anything.
3. Restarts the service and verifies the unit now launches <version>. On Windows, the restart also registers the service again from the launch command that step 1 read, so its registration matches what install writes. If the launched version does not match, the upgrade fails loudly rather than reporting a success it can’t stand behind.

If the launch target is under a node_modules folder that is not a global npm install (for example a local /srv/app/node_modules), the upgrade stops before it installs anything. The error names the unit’s node and the node that runs kici-admin. Use an archive upgrade, or install the package yourself and pass --restart-only.

No archive is downloaded, no versioned directory is created, and no symlink is flipped. The --from/--url archive flow (versioned directory, symlink, and rollback) is unchanged and remains a path for offline or air-gapped upgrades.

Why self-driving removes the runtime-mismatch footgun

Installing the package by hand (npm install -g) lands it under whichever node your shell’s PATH resolves — which, under a version manager (mise, nvm, asdf), is often not the pinned node the unit’s ExecStart points at. When they diverge, the unit keeps launching the old code even though the global package looks updated. The self-driving upgrade makes that impossible by construction: it installs with the unit’s own node and npm into the unit’s own prefix, so the copy it updates is exactly the copy the unit runs.

--restart-only for pre-staged installs

If you have already installed the package yourself under the unit’s runtime — an air-gapped host, a custom registry, or a bespoke provisioning flow — pass --restart-only to skip the self-driving install and just restart onto the already-installed package:

kici-admin orchestrator upgrade --restart-only --yes

In restart-only mode the CLI reads the version the unit will actually launch, verifies it matches the invoking kici-admin, and only then restarts. If the versions diverge (your manual install landed under a different runtime than the unit is pinned to), it aborts before stopping the service and names both versions and the launch path. On Windows, the restart also registers the service again.

--node to move to another Node.js runtime

A service that runs from an npm install uses the Node.js runtime that its unit names. To move it to another runtime, for example a new Node.js patch release, install that runtime first. Then pass its node executable with --node:

kici-admin orchestrator upgrade --version <version> --node /opt/node-v24.22.1/bin/node --yes
kici-admin agent upgrade --version <version> --node /opt/node-v24.22.1/bin/node --yes

The upgrade installs <package>@<version> with the npm of that runtime, into its own global prefix. Then it registers the service again to run the new node, and restarts it. On Linux and macOS, the new registration keeps the account that the service runs as. The old runtime and its copy of the package stay on disk unchanged.

Before it changes anything, the upgrade refuses a --node path that:

• is a version-manager shim, which starts the version manager instead of a fixed runtime. Pass the path that mise which node (or your version manager’s equivalent) prints.
• is not an executable file, or does not run --version.
• has no npm beside it (<prefix>/lib/node_modules/npm on Linux and macOS, <prefix>\node_modules\npm on Windows).

After the install and before the restart, the upgrade reads engines.node of the installed package. It refuses a runtime that the package does not accept. A path through a version-manager alias, such as mise node/24, gets a warning: the alias moves when you install another runtime, and the service moves with it.

After the restart, the upgrade checks that the unit launches the new node. It also moves a scheduled db backup unit of the orchestrator (kici-admin db backup --install-timer) to the new runtime. A bare-metal scaler whose binaryPath names the kici-agent of the old runtime gets a warning that names the kici-agent of the new runtime. Set binaryPath to that path in the scaler config.

With --restart-only, --node restarts onto a package that you already installed with the npm of the new runtime. --node works only for a service that runs from a global npm install. It is refused with --from, --url, --rollback, --pick, --cleanup and for a compose service. A full package carries its own Node.js, and a light package runs the Node.js version of its release.

--force for opaque launch targets

When the service was installed with an explicit --binary wrapper (or any launch target whose version can’t be read), there is no package metadata to resolve, so the self-driving install has no global package to update and the restart-only verification can’t run. In restart-only mode, passing --force bypasses the launch-target verification and restarts onto the installed package, leaving the manifest’s recorded version unchanged rather than guessing.

The infrastructure page surfaces the latest published version next to each node’s running version and reveals exactly this one-command upgrade when a newer version is available — see Monitoring & tracing.

What an upgrade does before it stops the service

An orchestrator upgrade takes two steps before anything stops:

1. A database dump. It writes pre-upgrade-<from>-<to>-<timestamp>.dump and its manifest into <instance-dir>/backups. Use --backup-dir <path> to put it elsewhere. If the dump cannot be taken — pg_dump missing, a version mismatch with the server, or an unresolvable database URL — the upgrade refuses and names the cause. --skip-backup proceeds without one; it prints a banner, because an upgrade with no dump has no recovery path.

2. A drain. It puts the coordinator into drain and waits for in-flight jobs to finish, up to --drain-timeout <seconds> (300 by default). On timeout it refuses and reports how many jobs are still running. --no-drain skips the wait and accepts that the restart fails them. After a successful upgrade no resume step is needed — the drain flag lives in memory, so the restarted process comes back accepting.

   A timeout refusal leaves the coordinator draining, so the jobs you are waiting for finish with nothing queueing behind them. It accepts no new work until the upgrade restarts it, or until you run kici-admin orchestrator resume.

   The drain needs the admin API. If the CLI has no --url / --token configured it cannot reach it, and the upgrade warns and continues rather than refusing.

Neither step applies to an agent upgrade: an agent has no database and no drain.

The bare-metal binaryPath check

Before an orchestrator upgrade installs or restarts anything, it reads the scaler config that the env file names (KICI_SCALER_CONFIG_PATH and KICI_SCALER_CONFIG_DIR). It then checks the binaryPath of each bare-metal label set. The expected agent is the kici-agent installed with the runtime of the service unit: the path that kici-admin orchestrator status prints as Agent binary.

• Not an executable file — the upgrade stops. Nothing is installed and the service keeps running, because the restarted orchestrator would not start. The message names the expected agent. Set binaryPath to that path and run the upgrade again. To upgrade anyway, pass --skip-binary-path-check: the upgrade then prints the problem as a warning and continues.
• A version-manager shim — the upgrade stops, as for a file that is not executable. A mise shim links to the mise executable, a Volta shim links to volta-shim, and an asdf or nodenv shim is a script in a shims folder. Each one runs the version manager, not the agent, so the restarted orchestrator refuses it at startup. The message names the expected agent, or the command that prints the file behind the shim, such as mise which kici-agent.
• An agent from another runtime — the upgrade continues with a warning. The warning names both paths and the version each one runs. The upgrade does not update that agent, so set binaryPath to the expected agent.
• A path through a version-manager alias — the upgrade continues with a warning. An example is the mise alias ~/.local/share/mise/installs/node/24/…. Such a path moves to another runtime when you install a new Node.js version, and that runtime has no kici-agent. The warning names a stable path to use.

An upgrade from an archive (--from / --url) does the same check, but it has no expected agent, so it does not report an agent from another runtime.

--rollback and --pick run the same check before they stop the service, but they only warn. They never stop for a binaryPath, because they are recovery paths. Read the warnings: a binaryPath that the check would refuse also stops the restarted orchestrator.

Rollback

To roll back the resolved instance to its previous version:

cd ~/kici-deploy
kici-admin orchestrator upgrade --rollback

This stops the service, switches the per-instance symlink to the previous version, and restarts. On Windows, it registers the service again for the previous version. It refuses a previous version from before KICI_ENV_FILE (see Windows) before it stops the service.

Take a dump first, and read the schema precondition below.

The schema precondition

A rollback moves the code back; it does not move the database back. Each release carries a fixed list of migrations. A version whose list lacks the names your database has already applied refuses to boot with corrupted migrations, and restarts in a loop. Auto-migration is on by default, so this happens at startup, unconditionally.

The upgrade checks for this. At every version change it records the migration head each installed version was running at. Re-running install keeps the recorded heads. Before the upgrade flips anything, it compares that head to the live database:

• Heads match — the switch proceeds.
• The database is ahead — the switch refuses, naming every migration applied since, and prints the two ways forward.
• No head recorded for the target — this instance was installed before the check existed, or the upgrade away from the target ran without admin access. So the head cannot be verified. The switch prints a prominent warning and falls through to the usual confirmation. A version change with admin access records the head, so the check is armed for the next one.

The upgrade reads the migration ledger through the admin API, as the drain does. Set KICI_ADMIN_TOKEN to an admin token, and KICI_ADMIN_URL to the address of the orchestrator. Without KICI_ADMIN_URL, kici-admin uses http://localhost:4000, the default port of an orchestrator on the same host. Without an admin token, the upgrade records no head, and a rollback prints that the schema was not checked, then falls through to the usual confirmation.

To roll back across a schema change, revert the schema first, while the newer version is still running:

kici-admin db backup
kici-admin db migrate --to <recorded-head>
kici-admin orchestrator upgrade --rollback

Or let the upgrade do it in one step:

kici-admin orchestrator upgrade --rollback --migrate-down

--migrate-down reverts the schema through the still-running newer service — the only binary that carries those migrations’ down() functions — and only then moves the version pointer. It is opt-in rather than automatic: a down() can be lossy, and running one unprompted while an operator is already handling an incident is not a default worth having.

If the switch does loop and you have no dump, start the older version with KICI_AUTO_MIGRATE=false. That boots only if every newer migration was purely additive.

Pick an installed version

To switch to a specific already-installed version (not just the immediately previous one), use --pick:

cd ~/kici-deploy
kici-admin orchestrator upgrade --pick

It lists every installed version, lets you choose one interactively (the active version is shown but not selectable), prints the change summary, and asks for confirmation before flipping the symlink / re-registering the service. Like --rollback, --pick only switches between versions already extracted under the instance’s install base — it never downloads. It requires an interactive terminal (for non-interactive switching, use --rollback or an explicit --from/--url archive).

--pick runs the same schema precondition as --rollback, and it matters more here: --pick can select any installed version, not only the one immediately before the current. On Windows, --pick also refuses a version from before KICI_ENV_FILE, as --rollback does.

Cleanup old versions

To remove old versions (keeps current and previous) of the resolved instance:

kici-admin orchestrator upgrade --instance-dir ~/kici-deploy --cleanup

Upgrade CLI flags

Flag	Description
--from <path>	Path to package archive (.tar.gz or .zip)
--url <url>	URL to download package archive from
--version <ver>	Target version string (e.g., 0.3.0). Required for archive upgrades and for the self-driving npm-source upgrade (the version to install)
--restart-only	npm-source upgrades: skip the self-driving install and just restart onto an already-installed package (pre-staged / air-gapped path)
--node <path>	npm-source upgrades: install the package with the npm of this Node.js executable and register the service again to run it (see --node)
--rollback	Roll back the resolved instance to its previous version
--pick	Interactively pick an already-installed version to activate (switch, no download). Requires an interactive terminal
--cleanup	Remove old versions of the resolved instance (keeps current and previous)
--force	Archive upgrades: overwrite an existing versioned directory. npm-source --restart-only: bypass launch-target version verification (for opaque --binary installs); restarts without recording a version
--yes	Skip confirmation prompt
--platform <type>	Force platform (systemd, launchd, windows, compose). Defaults to the platform in the install manifest
--instance-dir <path>	Deploy folder of the instance to upgrade
--name <name>	Service name (no default — must resolve via flag or CWD manifest)
--skip-backup	Skip the pre-upgrade database dump. Prints a banner; leaves the upgrade with no recovery path
--backup-dir <path>	Where the pre-upgrade dump goes (default <instance-dir>/backups)
--no-drain	Do not quiesce the coordinator first. In-flight jobs fail when the service restarts
--drain-timeout <s>	Seconds to wait for in-flight jobs to finish before refusing (default 300)
--migrate-down	On a rollback whose database is ahead, revert the schema to the target version’s recorded head first, through the still-running newer service

Database migrations during upgrade

The orchestrator auto-migrates its PostgreSQL database on startup (enabled by default). When the new version includes schema changes, migrations run automatically after the service restarts.

If you’ve disabled auto-migration (KICI_AUTO_MIGRATE=false), run migrations manually before you start the new binary:

kici-admin db migrate --database-url <url>   # Apply them directly; the orchestrator need not run
kici-admin db migrate                        # Or ask a running orchestrator to apply them

The order matters. A new orchestrator writes the columns its own release added, so starting it against an un-migrated database fails those writes at runtime rather than at boot. Nothing warns you at startup — the failure appears per request.

See Orchestrator setup — database for details on migration management.

Cluster upgrade order

In clustered deployments (coordinator + workers), nodes can be upgraded in any order as long as every node’s protocol version is at or above the minimum the others accept. Upgrade every node in the same window when a release raises the minimum accepted version — the release notes name it. Protocol 4 raises it from 3, so a node from 0.15.x or earlier (protocol 3) is refused by an upgraded peer until it is upgraded too. See Coordinator-worker — upgrade procedure for the full sequence.

Job recovery during upgrade

Running jobs are not lost during a planned upgrade. Agents reconnect within a 120-second grace period and resume in-flight work. See Job recovery for details on the recovery protocol and monitoring.

Uninstall procedure

kici-admin orchestrator uninstall [--instance-dir <path>] [--name <name>]

Uninstall:

• Resolves the target via the same --instance-dir / --name / CWD-manifest chain as the other lifecycle commands (refuses with a candidate list when no target can be resolved).
• Stops the service if running.
• Removes the service registration (systemd unit, launchd plist, or Windows service entry).
• Drops the host-wide index entry for the instance.
• Preserves config files, database, logs, and the on-disk manifest so subsequent commands can still reference the deploy folder if needed.

After uninstalling, you can manually clean up the resolved instance’s name-scoped directories:

# Linux (system)
sudo rm -rf /etc/kici/<name>/ /var/log/kici/<name>/

# Linux (user)
rm -rf ~/.config/kici/<name>/ ~/.local/share/kici/<name>/

# macOS (user)
rm -rf ~/Library/Application\ Support/kici/<name>/ ~/Library/Logs/kici/<name>/

# Windows
rmdir /s C:\ProgramData\kici\<name>

The instance index file (instances.json directly under each config root) and other instances’ directories are left in place.

Privilege model

The installer auto-detects privilege level:

• Root/admin — installs a system-wide service that runs at boot
• Regular user — installs a user-level service

For Linux user-level services, loginctl enable-linger is run automatically so the service survives logout.

Scaler privilege requirements

Some scaler configurations require elevated privileges:

Scaler	Minimum privilege
Container (Docker/Podman)	User (rootless) or root
Bare-metal	User
Firecracker	Root required

If you install a user-level service with Firecracker scaler configuration, the installer will warn you.

Restart policy

Services are configured with automatic restart on failure:

• Restart delay: systemd restarts a failed service after 1s, and launchd spaces restarts at least 1s apart. Windows increases the delay on each failure: 1s, 5s, 15s, then 30s.
• Retry limit: on systemd, 5 failed starts within 5 minutes stop the restarts. The service then stays stopped until you restart it manually. launchd has no retry limit. On Windows, the failure count resets after 5 minutes with no failure.

This is implemented via:

• systemd: Restart=on-failure and RestartSec in [Service]; the StartLimitBurst / StartLimitIntervalSec rate limit in [Unit] (systemd reads the start rate limit from [Unit] only)
• launchd: KeepAlive with SuccessfulExit set to false, and ThrottleInterval. launchd restarts the service when it exits non-zero or a signal kills it, and it has no retry limit. A plist installed by an older CLI sets KeepAlive to true, and launchd restarts that service after every exit.
• Windows: sc.exe failure with restart actions

The restart policy is written into the service definition at install time, so a service installed by an older CLI keeps whatever definition it was installed with. To apply the current restart policy to an existing instance, re-run install for it — on Unix, upgrade swaps the version symlink and does not regenerate the unit or plist:

kici-admin orchestrator install --name <service-name> --instance-dir /path/to/deploy-folder --env-file <env>

Pass the same --name and --instance-dir the instance was installed with — --name defaults to kici-orchestrator, so omitting it on a differently-named instance registers a second service instead of refreshing the existing one.

Confirm the window systemd actually applied (drop --user for a system-level install):

systemctl --user show -p StartLimitIntervalUSec -p StartLimitBurst <service-name>
# StartLimitIntervalUSec=5min
# StartLimitBurst=5

The services distinguish clean stops from fatal failures by exit code: an intentional shutdown (SIGTERM/SIGINT from a service stop, or an admin-initiated drain) exits 0 and does not trigger a restart, while a fatal internal error (an uncaught exception) runs the same graceful teardown but exits non-zero so the on-failure restart policy brings the service back automatically. This holds on launchd too, except for a plist from an older CLI (see above), which restarts the service after any exit until stop unloads the job.

Firecracker scaler setup

When the orchestrator is configured to use the Firecracker scaler, the host needs additional setup. The installer does not do it. It only warns when the env file names Firecracker and you install a user-level service.

• The orchestrator creates the network bridge and its NAT rules itself at startup, unless the scaler config sets firecracker.autoProvisionHost: false.
• Everything else is a one-time host setup: KVM, the Firecracker and jailer binaries, the jailer user and directories, a guest kernel, and the agent rootfs image. The setup scripts for these steps, and the rootfs build script, are in scripts/firecracker/ of the source repository, github.com/kici-dev/kici-public, from KiCI 0.12.0 on.
• The scaler’s network and jailer operations need root privileges, so run the installer as root to install a system-level service. Firecracker host setup, Step 10 describes the non-root options for a service you define yourself.

Follow Firecracker host setup for every step, and the Firecracker rootfs build guide for the image.

Troubleshooting

Service won’t start

1. Check the env file exists and has valid configuration:

   # Per-instance env path is name-scoped: /etc/kici/<name>/<name>.env
   cat /etc/kici/kici-orchestrator/kici-orchestrator.env

2. Check logs for errors:

   kici-admin orchestrator logs --instance-dir ~/kici-deploy --no-follow --since 5m --level error

3. Verify database connectivity:

   kici-admin diagnose

“No instance specified and no manifest in CWD”

This refusal means the lifecycle command could not resolve a target. Either cd into the deploy folder that contains the manifest, or pass --instance-dir <path> / --name <name> explicitly. The error message lists every installed instance on the host with its deploy folder; use one of those values.

”exists in the init system but has no manifest”

--name found the unit in the init system but could not locate its deploy folder. This happens only for a unit installed before the deploy folder was embedded in the service definition, when the instance index has also been lost. Recover by passing --instance-dir <deploy folder> once — for an upgrade, kici-admin orchestrator upgrade --instance-dir <path> regenerates the unit with the embedded marker, after which --name resolves on its own and the index self-heals. Units installed normally embed the marker, so they recover automatically even with an empty index.

Permission denied

• System-level operations require root/admin
• Ensure the service user has read access to the config file
• Firecracker scaler requires root — don’t install as user-level

Port already in use

Change the port in the env file (KICI_PORT=4001) and restart:

kici-admin orchestrator restart

User-level service dies on logout (Linux)

Ensure linger is enabled:

loginctl enable-linger $(whoami)

This is done automatically during install, but can be undone if system settings change.

Windows service fails to start

1. Check Windows Event Log: Event Viewer > Windows Logs > Application
2. Verify shawl was downloaded: check %LOCALAPPDATA%\kici\deps\ or C:\ProgramData\kici\deps\
3. Ensure the binary path doesn’t contain spaces without proper quoting
4. Read the service log in C:\ProgramData\kici\{name}\logs\. The line [kici] KICI_ENV_FILE names <path>, which could not be read (<code>) means that the env file at that path is missing, or that the service account cannot read it. Put the file back, then re-run install --no-wizard as the Windows quick start describes, to reset its ACL.