kici-admin: cluster & infrastructure

Guide

orchestrator — service lifecycle

kici-admin orchestrator install [--wizard] [--platform systemd|launchd|windows|compose] [--env-file <path>] [--binary <path>] [--dev] [--name <name>] [--instance-dir <path>] [--force]
kici-admin orchestrator uninstall [--platform <type>] [--instance-dir <path>] [--name <name>]
kici-admin orchestrator start [--platform <type>] [--instance-dir <path>] [--name <name>]
kici-admin orchestrator stop [--platform <type>] [--instance-dir <path>] [--name <name>]
kici-admin orchestrator restart [--platform <type>] [--instance-dir <path>] [--name <name>]
kici-admin orchestrator status [--platform <type>] [--instance-dir <path>] [--name <name>] [--json]
kici-admin orchestrator logs [--platform <type>] [--instance-dir <path>] [--name <name>] [--since <duration>] [--level <level>] [--json] [--no-follow]
kici-admin orchestrator upgrade [--from <path>] [--url <url>] [--version <version>] [--cleanup] [--rollback] [--pick] [--yes] [--force] [--platform <type>] [--instance-dir <path>] [--name <name>]

Manages the orchestrator as a native system service. The install --wizard flow handles database setup, encryption key generation, Platform credentials, and optionally adding your first source. Lifecycle targeting is folder-anchored — see Service installation guide for platform-specific details and the full description of the manifest, the instance index, and the name-scoped on-disk layout.

The upgrade command uses a name-scoped versioned directory layout: new versions are extracted under the resolved instance’s own <installBase>/<name>/ tree alongside old ones, and a per-instance symlink is atomically switched. Other installed instances on the host are not touched. Use --rollback to revert to the previous version and --cleanup to remove old versions (keeping current and previous). Use --pick to switch to any already-installed version: it lists every installed version, lets you choose one interactively (the active version is shown but not selectable), prints the change summary, and confirms before switching. Like --rollback, --pick only switches between versions already extracted under the instance’s install base — it never downloads.

Every lifecycle command (uninstall, upgrade, start, stop, restart, status, logs) resolves its target through the priority chain --instance-dir > --name > manifest in the current working directory. A bare kici-admin orchestrator <cmd> outside any deploy folder with no flags refuses non-zero and prints the candidate list of installed orchestrator instances on the host.

cluster — cluster identity recovery (direct DB + S3)

kici-admin cluster reconcile-identity [--adopt-db] [--dry-run] [--yes] [--database-url <url>] [--bucket <bucket>] [--prefix <prefix>] [--region <region>] [--endpoint <url>] [--force-path-style]

Reconciles the orchestrator’s cluster_meta.cluster_id with the durable S3 sentinel — the cross-restart / peer anchor that lets a redeployed orchestrator reclaim its identity. Talks directly to the orchestrator Postgres and the same S3 bucket the running process uses (no HTTP admin path), so it works while the orchestrator is offline. Requires AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for the sentinel.

Default direction restores the DB from the sentinel (db-from-sentinel) — the common recovery after a database was rebuilt but the durable identity in object storage is authoritative.
--adopt-db reverses the direction (sentinel-from-db): rewrite the sentinel from the DB’s current cluster_id.
--dry-run reports drift and exits without changing anything.
--yes skips the confirmation prompt and applies on drift (scripted use).
Storage flags fall back to the orchestrator’s KICI_DATABASE_URL / KICI_STORAGE_BUCKET / KICI_STORAGE_PREFIX (default empty = bucket root) / KICI_STORAGE_REGION / KICI_STORAGE_ENDPOINT when omitted. --force-path-style selects S3 path-style addressing.

cluster-name — orchestrator cluster identity

kici-admin cluster-name get [--format json|table]
kici-admin cluster-name set <name> [--format json|table]

Manages this orchestrator’s human-friendly cluster name — the identifier that surfaces on Platform’s connection registry and in the dashboard’s per-orch URL segment.

get prints the current cluster name plus a looksAutoGenerated flag indicating whether it’s still the default placeholder.
set <name> renames the cluster. Mutating it requires admin access (RBAC secret.write). The response reports the prior value and a reconnectRequired flag — restart the orchestrator (or run kici-admin orchestrator restart) to publish the new name to Platform.

Talks to the orchestrator admin API directly (not the Platform dashboard proxy) so the CLI stays operable even when Platform is unavailable.

cluster-settings — fleet-wide runtime tunables

kici-admin cluster-settings show [--format json|table]
kici-admin cluster-settings set --<knob> <value> [--<knob> <value> ...]
kici-admin cluster-settings reset [--<knob> ...]

Reads and writes the cluster-global tunables — one row shared by every orchestrator in the cluster. These are the knobs with no per-tenant meaning: ingest limits applied before an org is resolved, process-global singletons, and leader-only sweeper timings. Per-tenant knobs live in org-settings instead.

show prints every knob with its effective value; a knob left unset shows as null, meaning the cluster default from the orchestrator’s own configuration applies.
set requires at least one knob flag and takes several in a single call. Each numeric knob enforces a floor (the same floor the admin route validates against), so a value below it is rejected rather than silently clamped.
reset clears the named knobs back to null (the configured default). Passing no knob flag clears every knob at once.

Changing a knob takes effect on the running cluster without a restart — the orchestrator reads these values live at each operation. See Cluster settings for the full per-knob table, including what each one governs and its default.

scaler — scaler maintenance (local, no orchestrator)

kici-admin scaler reap-orphans [--config <path>] [--force] [--json]

Frees leaked Firecracker / container resources (orphaned microVMs, TAP devices, containers) without a running orchestrator. Runs locally against the host using the orchestrator config, so it is the recovery path when the orchestrator crashed and left scaler-managed resources behind.

By default the command refuses to reap when a local orchestrator reports healthy (so it never races a live process); pass --force to override.

firecracker — host networking provisioning

kici-admin firecracker provision [--bridge <name>] [--cidr <cidr>] [--table <name>] [--host-iface <iface>] [--persist] [--sudo]
kici-admin firecracker verify [--bridge <name>] [--cidr <cidr>] [--table <name>] [--sudo]
kici-admin firecracker teardown [--bridge <name>] [--cidr <cidr>] [--table <name>] [--sudo]

Provisions and verifies the host-side bridge interface + NAT/egress-isolation rules a Firecracker scaler needs. These commands run on the Firecracker host (not against the orchestrator HTTP API) and typically require root — pass --sudo to wrap the privileged steps with sudo -n on a non-root host.

provision creates or heals a host bridge with a gateway address, NAT egress, and an nftables table. --cidr sets the gateway IP + prefix (e.g. 10.0.0.1/24); --host-iface names the NAT egress interface (auto-detected when omitted). Pass --persist to install a systemd oneshot so the bridge survives a reboot.
verify checks that the named bridge is up with its address and nft table present. Use it after provision (or in a health check) to confirm host networking.
teardown removes the bridge interface and its nft table. It deliberately leaves the NetworkManager unmanaged-interface conf file in place, because that file is host-scoped and protects every kici-* interface on the host — removing it would let NetworkManager adopt the other bridges and strip their gateway IPs.

See Firecracker host setup and the Firecracker scaler backend for the full host-networking walkthrough.

Reference

`kici-admin cluster`

Cluster identity recovery (DB <-> S3 sentinel reconcile).

Synopsis: kici-admin cluster

`kici-admin cluster reconcile-identity`

Reconcile cluster_meta.cluster_id with the S3 sentinel. Default restores the DB from the sentinel.

Synopsis: kici-admin cluster reconcile-identity [options]

Options

Option	Default	Description
`--database-url <url>`		Orchestrator DB URL (else KICI_DATABASE_URL)
`--bucket <bucket>`		S3 bucket (else KICI_STORAGE_BUCKET)
`--prefix <prefix>`		Storage prefix (else KICI_STORAGE_PREFIX, default empty = bucket root)
`--region <region>`		S3 region (else KICI_STORAGE_REGION)
`--endpoint <url>`		S3 endpoint (else KICI_STORAGE_ENDPOINT)
`--force-path-style`		Use S3 path-style addressing
`--adopt-db`		Reverse direction: rewrite the sentinel from the DB cluster_id
`--dry-run`		Report drift and exit without changing anything
`--yes`		Skip confirmation and apply on drift

`kici-admin cluster-name`

Manage this orchestrator’s cluster name (Platform-visible identifier)

Synopsis: kici-admin cluster-name

`kici-admin cluster-name get`

Print the current cluster name.

Synopsis: kici-admin cluster-name get [options]

Options

Option	Default	Description
`--format <format>`	`table`	Output format: json\|table

`kici-admin cluster-name set`

Rename the cluster. Cluster name must match ^[a-z][a-z0-9-]{0,62}$ (lowercase letters, digits, hyphens; start with a letter; ≤63 chars).

Synopsis: kici-admin cluster-name set <name> [options]

Arguments

Argument	Required	Variadic	Description
`name`	yes	no

Options

Option	Default	Description
`--format <format>`	`table`	Output format: json\|table

`kici-admin cluster-settings`

Manage the fleet-wide cluster tunables (null = cluster default)

Synopsis: kici-admin cluster-settings

`kici-admin cluster-settings reset`

Clear cluster-global overrides (all, or the named knobs) back to cluster defaults

Synopsis: kici-admin cluster-settings reset [options]

Options

Option	Default	Description
`--format <format>`	`table`	Output format: json\|table
`--max-github-payload-bytes`		Clear only Max GitHub payload bytes
`--event-log-max-payload-bytes`		Clear only Event-log max payload bytes
`--lock-file-max-bytes`		Clear only Lock-file max bytes
`--webhook-dedup-ttl-ms`		Clear only Webhook dedup TTL (ms)
`--contributor-cache-ttl-ms`		Clear only Contributor-cache TTL (ms)
`--event-router-event-ttl-seconds`		Clear only Event-router event TTL (s)
`--event-router-max-dispatch-attempts`		Clear only Event-router max dispatch attempts
`--queue-max-depth`		Clear only Queue max depth
`--reroute-flap-grace-ms`		Clear only Reroute flap grace (ms)
`--max-fanout-hosts`		Clear only Max fanout hosts
`--event-router-rate-limit-per-workflow-per-minute`		Clear only Event-router rate limit (/wf/min)
`--cache-max-tarball-bytes`		Clear only Cache max tarball bytes
`--cache-ttl-days`		Clear only Cache TTL (days)
`--check-run-tracking-ttl-days`		Clear only Check-run tracking TTL (days)
`--concurrency-wait-timeout-ms`		Clear only Concurrency wait timeout (ms)
`--agent-token-ttl-ms`		Clear only Agent token TTL (ms)
`--ownership-db-check-timeout-ms`		Clear only Ownership DB check timeout (ms)
`--unroutable-grace-ms`		Clear only Unroutable fast-fail grace (ms)
`--dashboard-verified-issuer`		Clear only Dashboard verified issuer

`kici-admin cluster-settings set`

Set one or more cluster-global tunables. At least one knob flag required.

Synopsis: kici-admin cluster-settings set [options]

Options

Option	Default	Description
`--format <format>`	`table`	Output format: json\|table
`--max-github-payload-bytes <value>`		Max GitHub payload bytes (integer >= 1024)
`--event-log-max-payload-bytes <value>`		Event-log max payload bytes (integer >= 1024)
`--lock-file-max-bytes <value>`		Lock-file max bytes (integer >= 1024)
`--webhook-dedup-ttl-ms <value>`		Webhook dedup TTL (ms) (integer >= 1000)
`--contributor-cache-ttl-ms <value>`		Contributor-cache TTL (ms) (integer >= 1000)
`--event-router-event-ttl-seconds <value>`		Event-router event TTL (s) (integer >= 1)
`--event-router-max-dispatch-attempts <value>`		Event-router max dispatch attempts (integer >= 1)
`--queue-max-depth <value>`		Queue max depth (integer >= 1)
`--reroute-flap-grace-ms <value>`		Reroute flap grace (ms) (integer >= 1000)
`--max-fanout-hosts <value>`		Max fanout hosts (integer >= 1)
`--event-router-rate-limit-per-workflow-per-minute <value>`		Event-router rate limit (/wf/min) (integer >= 1)
`--cache-max-tarball-bytes <value>`		Cache max tarball bytes (integer >= 1024)
`--cache-ttl-days <value>`		Cache TTL (days) (integer >= 1)
`--check-run-tracking-ttl-days <value>`		Check-run tracking TTL (days) (integer >= 0)
`--concurrency-wait-timeout-ms <value>`		Concurrency wait timeout (ms) (integer >= 1000)
`--agent-token-ttl-ms <value>`		Agent token TTL (ms) (integer >= 1000)
`--ownership-db-check-timeout-ms <value>`		Ownership DB check timeout (ms) (integer >= 100)
`--unroutable-grace-ms <value>`		Unroutable fast-fail grace (ms) (integer >= 0)
`--dashboard-verified-issuer <value>`		Dashboard verified issuer (an absolute http(s) URL)

`kici-admin cluster-settings show`

Print the current cluster-global tunables

Synopsis: kici-admin cluster-settings show [options]

Options

Option	Default	Description
`--format <format>`	`table`	Output format: json\|table

`kici-admin firecracker`

Provision and verify Firecracker host networking

Synopsis: kici-admin firecracker

`kici-admin firecracker provision`

Create/heal a Firecracker host bridge (NAT + egress isolation)

Synopsis: kici-admin firecracker provision [options]

Options

Option	Default	Description
`--bridge <name>`		bridge interface name (e.g. kici-br0)
`--cidr <cidr>`		gateway IP + prefix (e.g. 10.0.0.1/24)
`--table <name>`	`kici`	nft table name
`--host-iface <iface>`		NAT egress interface (auto-detected if omitted)
`--persist`		install a systemd oneshot so the bridge survives reboot
`--sudo`		wrap privileged commands with sudo -n (non-root host)

`kici-admin firecracker teardown`

Remove a Firecracker host bridge + its nft table (leaves NM conf in place)

Synopsis: kici-admin firecracker teardown [options]

Options

Option	Default	Description
`--bridge <name>`		bridge interface name
`--cidr <cidr>`	`0.0.0.0/0`	gateway IP + prefix (unused but accepted for symmetry)
`--table <name>`	`kici`	nft table name
`--sudo`		wrap privileged commands with sudo -n

`kici-admin firecracker verify`

Check a Firecracker host bridge is up with its addr + nft table

Synopsis: kici-admin firecracker verify [options]

Options

Option	Default	Description
`--bridge <name>`		bridge interface name
`--cidr <cidr>`		gateway IP + prefix
`--table <name>`	`kici`	nft table name
`--sudo`		wrap privileged commands with sudo -n

`kici-admin orchestrator`

Manage orchestrator service installation and lifecycle

Synopsis: kici-admin orchestrator

`kici-admin orchestrator drain`

Quiesce this coordinator before upgrading (stop dispatching new jobs)

Synopsis: kici-admin orchestrator drain [options]

Options

Option	Default	Description
`--wait`		Block until in-flight jobs finish (jobsRunning reaches 0)
`--timeout <seconds>`	`300`	Max seconds to wait with —wait
`--status`		Report drain status without changing it

`kici-admin orchestrator install`

Install the orchestrator as a system service

Synopsis: kici-admin orchestrator install [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd, launchd, windows, compose)
`--env-file <path>`		Path to existing env/config file to use
`--binary <path>`		Path to orchestrator binary (default: current executable)
`--dev`		Dev mode: spin up PostgreSQL container on port 15432
`--wizard`		Run the interactive setup wizard (default on an interactive terminal)
`--no-wizard`		Skip the wizard and write a stub env file to edit by hand
`--name <name>`	`kici-orchestrator`	Service name
`--system`		Install as system-level service (requires root)
`--user-level`		Install as user-level service (no root required)
`--user <name>`		Run the service as the named user (system-level launchd only; sets UserName in plist so the daemon drops privileges)
`--instance-dir <path>`		Deploy folder; the instance manifest is written here (default: current working directory)
`--force`		Overwrite an existing same-named foreign instance

`kici-admin orchestrator logs`

Tail and follow orchestrator service logs

Synopsis: kici-admin orchestrator logs [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd\|launchd\|windows\|compose)
`--instance-dir <path>`		Deploy folder of the instance whose logs to read
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service
`--since <duration>`		Show logs since duration (e.g. 1h, 30m)
`--level <level>`		Filter by log level (error\|warn\|info)
`--json`		Output as structured JSON
`--no-follow`		Snapshot mode (do not tail)

`kici-admin orchestrator restart`

Restart the orchestrator service

Synopsis: kici-admin orchestrator restart [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd, launchd, windows, compose)
`--instance-dir <path>`		Deploy folder of the instance to restart
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service

`kici-admin orchestrator resume`

Resume dispatching new jobs (undo a drain)

Synopsis: kici-admin orchestrator resume

`kici-admin orchestrator start`

Start the orchestrator service

Synopsis: kici-admin orchestrator start [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd, launchd, windows, compose)
`--instance-dir <path>`		Deploy folder of the instance to start
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service

`kici-admin orchestrator status`

Show orchestrator service status and health information

Synopsis: kici-admin orchestrator status [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd\|launchd\|windows\|compose)
`--instance-dir <path>`		Deploy folder of the instance to inspect
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service
`--json`		Output as JSON

`kici-admin orchestrator stop`

Stop the orchestrator service

Synopsis: kici-admin orchestrator stop [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd, launchd, windows, compose)
`--instance-dir <path>`		Deploy folder of the instance to stop
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service

`kici-admin orchestrator uninstall`

Remove the orchestrator service registration

Synopsis: kici-admin orchestrator uninstall [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd, launchd, windows, compose)
`--instance-dir <path>`		Deploy folder of the instance to uninstall
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--system`		Operate against the system-level service (requires root)
`--user-level`		Operate against the user-level service

`kici-admin orchestrator upgrade`

Upgrade orchestrator to a new version using versioned directory layout

Synopsis: kici-admin orchestrator upgrade [options]

Options

Option	Default	Description
`--platform <type>`		Service platform (systemd\|launchd\|windows\|compose)
`--instance-dir <path>`		Deploy folder of the instance to upgrade
`--name <name>`		Service name (no default — must resolve via flag/CWD)
`--from <path>`		Path to package archive (.tar.gz or .zip)
`--url <url>`		URL to download package archive from
`--version <version>`		Target version string (e.g., 0.3.0)
`--yes`		Skip confirmation prompt
`--force`		Overwrite existing versioned directory
`--cleanup`		Remove old versions (keeps current and previous)
`--rollback`		Roll back to the previous version
`--pick`		Interactively pick an installed version to activate
`--restart-only`		Restart the already-installed package without installing (skip self-drive)
`--no-agent-packages`		Skip auto-producing + uploading the fleet agent payloads for the new version
`--agent-package-platforms <list>`		Override the fleet platform set to (re)package: single \| CSV \| all (default: discover from the cache bucket)
`--node-mirror <url>`		nodejs.org mirror override for the agent package build
`--npm-registry <url>`		npm registry override for the agent package build

`kici-admin scaler`

Scaler maintenance (local, no orchestrator)

Synopsis: kici-admin scaler

`kici-admin scaler reap-orphans`

Free leaked Firecracker/container resources without a running orchestrator

Synopsis: kici-admin scaler reap-orphans [options]

Options

Option	Default	Description
`--config <path>`		Path to the orchestrator config (default: KICI_CONFIG or /etc/kici/orchestrator.yaml)
`--force`	`false`	Reap even if the local orchestrator reports healthy
`--json`	`false`	Emit machine-readable JSON counts