Skip to content
KiCI Docs

Firecracker data disk recovery

Symptom

Section titled “Symptom”

A Firecracker scaler node — especially a rootless edge worker — can fill its data disk with leaked jailer chroots. When the disk reaches 100%, the orchestrator can no longer write its log and database files and crash-loops at startup with:

ENOSPC: no space left on device, write

The systemd unit sits in activating (auto-restart) and never reaches a healthy state. The data volume (the scaler’s chrootBaseDir, default /srv/jailer) is full of leftover per-VM chroot directories under firecracker/.

This is a bootstrap deadlock: the in-process orphan sweep that would free the disk only runs once the orchestrator is up, but the orchestrator cannot start while the disk is full.

Recovery

Section titled “Recovery”

Run the standalone reaper on the affected host:

Terminal window
kici-admin scaler reap-orphans

The command loads the orchestrator’s local config, reconstructs the scaler backends without a running orchestrator or database, and frees leaked resources. It is the supported recovery path — no manual chown / rm of chroot directories is needed.

What it does:

  • Reaps only dead VMs. A liveness pre-scan reads each chroot’s firecracker.pid and protects any VM whose process is still alive. Chroots and TAP devices belonging to dead VMs are removed; live VMs are never touched.
  • Reclaims ownership first. On rootless nodes the reaper reclaims ownership of each leaked chroot before deleting it, so disk owned by the jailer’s subuid is actually freed.
  • Also sweeps container scalers. Once the health gate has passed, every container scaler in the config is swept too. That pass removes the stopped containers stamped with the scaler’s own kici-scaler-name label. A running container is left alone: the standalone command has no orchestrator registry to consult, so it treats every running agent as live. Containers another scaler or another orchestrator stamped are never touched.
  • No-ops while healthy. The command probes the local orchestrator health endpoint. If the orchestrator is up and healthy, the command prints a notice and exits without doing anything — the running orchestrator already reaps its own orphans. Pass --force to reap anyway.

Useful flags:

  • --force — skip the health gate and reap even if the orchestrator reports healthy. Use it only on a node you know is wedged: a healthy orchestrator already reaps its own orphans, so a forced run races it for the same stopped containers and dead chroots.
  • --config <path> — point at a non-default orchestrator config location (also honoured via the KICI_CONFIG environment variable).
  • --json — emit machine-readable counts, for scripting and host timers.

Automatic recovery

Section titled “Automatic recovery”

Startup disk-space guard. Before opening its log and database handles, the orchestrator checks free space on the chroot volume. If it is below the threshold, it reaps Firecracker orphans inline and continues startup only if enough space was freed. If the reap cannot free enough, the orchestrator logs a single actionable line naming kici-admin scaler reap-orphans instead of crash-looping opaquely on a buffered write.

This guard is the only recovery the orchestrator performs on its own, and it only runs at startup. A node that fills its disk between restarts stays full until something reaps it.

Section titled “Recommended: a host timer as a backstop”

kici-admin scaler reap-orphans is built to be safe on a schedule, but nothing schedules it for you — set up a host timer yourself on each Firecracker node, especially a rootless edge peer you do not watch closely. Every 30 minutes is a reasonable cadence.

Two properties make a periodic run safe:

  • It probes the local orchestrator health endpoint first, so on a healthy node it prints a notice and exits 0 without touching anything.
  • The Firecracker reap is liveness-driven, so even when it does run it only removes chroots and TAP devices of dead VMs.

Use --json so the timer’s journal records what each run freed. Do not add --force to a scheduled run: it skips the health gate, and a run that overlaps a healthy orchestrator’s own sweep does no useful work.

See also

Section titled “See also”