Auto-scaler: Firecracker backend

The Firecracker backend provisions agents as ephemeral KVM-backed microVMs. Each job runs in a dedicated VM with hardware-level isolation, sub-125 ms boot times, and automatic cleanup — the strongest isolation model KiCI supports. For fields shared across all backends, see Common configuration. For host setup, see the Firecracker host setup.

When to choose Firecracker

• Security isolation: Hardware-level isolation via KVM. No shared kernel with the host.
• Untrusted workloads: Safe for running CI jobs from public repositories or untrusted contributors.
• Compliance: Meets requirements for workload isolation in regulated environments.

What isolates one job from another

Three mechanisms, each covering a boundary the others do not:

• The KVM boundary. Each job runs in its own microVM with its own kernel. A job cannot read another job’s memory or filesystem.
• Bridge port isolation. Every VM’s TAP device is enslaved to the bridge with isolated on, so a VM can reach the gateway — and through it, the internet — but cannot reach another VM on the same bridge. Without it, VM traffic is switched at layer 2 and no packet filter ever sees it. The orchestrator refuses to start if the host kernel rejects the flag.
• The nftables chain. Per-VM rules matching the VM’s own address deny RFC1918 ranges and the cloud-metadata address, and apply the label set’s networkPolicy. They sit ahead of the host baseline, which the chain reaches through a jump as its last rule — so a per-VM denyAll or allowlist decides the packet before the baseline can.

For how Firecracker compares to the container and bare-metal backends for confining customer workflow code, see Agent execution security.

Network configuration

The top-level firecracker: key defines VM networking, shared across all Firecracker scalers:

firecracker:
  cidr: '10.0.0.0/24' # CIDR range for VM IP allocation. Default: '10.0.0.0/24'.
  bridgeName: 'kici-br0' # Host bridge interface name. Default: 'kici-br0'.
  gateway: '10.0.0.1' # Gateway IP (assigned to the bridge). Default: '10.0.0.1'.
  netmask: '255.255.255.0' # Subnet mask for guest networking. Default: '255.255.255.0'.
  table: 'kici' # nftables table name for this host bridge (disjoint per bridge). Default: 'kici'.
  autoProvisionHost: true # Verify + provision this host bridge on startup (self-heal). Default: true.

With autoProvisionHost at its default (true), the orchestrator verifies and, if needed, provisions this host bridge when it starts — a fresh host needs no manual kici-admin firecracker provision step. Set it false to keep explicit operator control and provision the host network yourself. See Firecracker host setup for the full behavior and the manual/--persist opt-out flow.

Firecracker-specific fields

Scaler-level fields:

• firecrackerPath — Path to the Firecracker binary. Required.
• jailerPath — Path to the jailer binary. Required.
• kernelPath — Default kernel path. Required.
• chrootBaseDir — Jailer chroot base directory. Optional; default /srv/jailer.
• uid / gid — Jailer UID / GID. Required.
• vcpuCount — Default vCPU count for VMs. Optional; default 2.
• memSizeMib — Default memory in MiB for VMs. Optional; default 512.
• requireSudo — Wrap the privileged commands the backend runs (ip, chown, chmod, and nft for per-VM network isolation) with sudo -n. Optional; default false. Set it true when the orchestrator runs as a non-root user (for example a user-mode systemd unit) and the operator has a NOPASSWD sudoers entry for those binaries. Leave it unset when the orchestrator is root or already holds the required capabilities — -n fails fast rather than prompting, so an unnecessary true turns a working setup into a spawn failure.

Label-set-level fields:

• rootfsPath — Path to a pre-built ext4 rootfs image. Required on every Firecracker label set.
• kernelPath — Override the scaler-level kernel path for this label set. Optional.
• vcpuCount / memSizeMib — Override the scaler-level VM CPU / memory for this label set. Optional.
• overlayDriveSizeMib — Copy-on-write overlay drive size in MiB. Optional; default 2048.

Configuration

version: 1
globalMaxAgents: 50

firecracker:
  cidr: '10.0.0.0/24'
  bridgeName: kici-br0
  gateway: '10.0.0.1'
  netmask: '255.255.255.0'

scalers:
  - name: fc-linux
    type: firecracker
    maxAgents: 20
    firecrackerPath: /usr/local/bin/firecracker
    jailerPath: /usr/local/bin/jailer
    kernelPath: /var/lib/kici/vmlinux
    chrootBaseDir: /srv/jailer # Optional, default: /srv/jailer
    uid: 10000
    gid: 10000
    vcpuCount: 2
    memSizeMib: 512
    orchestratorUrl: 'ws://10.0.0.1:8080/ws'
    labelSets:
      - labels: [linux, vm]
        rootfsPath: /var/lib/kici/rootfs-alpine.ext4

Key differences from container/bare-metal:

• The firecracker top-level key defines global network configuration (CIDR pool, bridge name).
• Scaler-level fields include firecrackerPath, jailerPath, kernelPath, chrootBaseDir, uid, gid.
• Each label set requires a rootfsPath pointing to a pre-built ext4 image.
• orchestratorUrl should point to the bridge gateway IP (VMs cannot reach localhost). It resolves in priority order: the scaler-level orchestratorUrl field, then the KICI_ORCHESTRATOR_URL environment variable, then the default ws://127.0.0.1:<orchestrator port>/ws (the orchestrator’s own KICI_PORT, 4000 unless you changed it).

Sizing memory

The memSizeMib default of 512 boots the agent. It does not run a job. A job installs its .kici/ dependencies inside the VM, and that install is the peak. At 1024 MiB the install fails after several minutes with out-of-memory kills. The failure looks like a hang, not an error. Give a VM that runs jobs 2048 MiB or more. Raise it further for a workflow with a large dependency tree.

Set it per label set when only some of your jobs are heavy, so a small VM still serves the light ones.

DB migration

The Firecracker backend requires the ip_allocations PostgreSQL table for DB-backed IP allocation. This is created by migration 001_initial and runs automatically on orchestrator startup.

MMDS credential model

Firecracker VMs use a hybrid credential model to prevent customer workflow code from reading orchestrator credentials:

1. Boot: The orchestrator URL, agent ID, labels, scaler-managed flag, gateway IP, and optionally an auth token and backpressure mode are injected via MMDS metadata at VM startup
2. Registration: The agent connects via WebSocket and sends agent.register
3. Config delivery: The orchestrator replies with register.ack containing the agent’s confirmed config (labels, max concurrent jobs, scaler-managed flag)
4. Agent-side blocking: After receiving register.ack, the agent blocks MMDS access via iptables -A OUTPUT -d 169.254.169.254 -j DROP
5. Agent acknowledgment: The agent sends config.ack to confirm it received and applied the config
6. Host-side clearing: The orchestrator clears MMDS data via the Firecracker API after receiving config.ack

This two-sided approach (agent blocks + orchestrator clears) ensures MMDS data is inaccessible to customer code even if one side fails. The MMDS contains only agent bootstrap data (orchestrator URL, agent ID, labels including auto-injected kici:agent:*, kici:scaler:*, and kici:role:* labels, scaler-managed flag, gateway IP, optionally an ephemeral agent token, optionally a backpressure mode, and optionally KICI_AGENT_ENV_*-forwarded env vars under meta-data/kici-env/) — no long-lived API keys or secrets.

Helper scripts

KiCI provides host-setup tooling — kici-admin firecracker for networking, plus helper scripts in scripts/firecracker/:

Tool	Purpose
validate.sh	Check host prerequisites (KVM, binaries, network)
kici-admin firecracker provision	Create bridge interface + NAT rules; --persist for reboot survival
jailer-setup.sh	Prepare jailer directory structure and cgroups
rootfs-builder.sh	Convert Docker images to ext4 rootfs

For the complete setup guide, see Firecracker host setup.