Orchestrator monitoring pack

KiCI ships a starter monitoring pack for a self-hosted orchestrator: one importable Grafana dashboard and a set of Prometheus alert rules covering the signals that matter at 2am — dispatch queue depth, stale runs, the event DLQ, scaler spawn failures, and agent-vs-backlog health. Every metric the pack references is one the orchestrator already exports on its /metrics endpoint, so there is no extra instrumentation to install.

What’s in the pack

• Grafana dashboard — a single “KiCI orchestrator health” board. It shows fleet size, dispatch queue by status and runner label, and the event DLQ with delivery outcomes. It also shows scaler CPU/memory reservation, spawn failures, and the event-scaler reaper guards. The last rows show consecutive job failures, declared-host reachability, database collation drift, inbound webhook rate, org trust-policy gate decisions, state-replay breaker trips, and warm-pool fill.
• Alert rules — Prometheus alert rules with tuned thresholds and per-rule rationale. Each one is listed with its expression and threshold rationale below.

Import the dashboard

1. In Grafana, go to Dashboards → New → Import.
2. Upload kici-orchestrator-dashboard.json (or paste its contents).
3. When prompted, select your Prometheus data source for the DS_PROMETHEUS input.
4. Save.

The dashboard prompts for the data source on import rather than hard-coding one, so it drops into any Grafana that scrapes your orchestrator.

Install the alert rules

Wire kici-orchestrator-alerts.yaml into your Prometheus rule_files: (or your Grafana / Mimir ruler) and reload. Point the alerts at your notification channel of choice. The file is a standard Prometheus rule-group document — a single kici-orchestrator group — so it needs no conversion.

Threshold rationale

Thresholds are starting points. QueueDepthSaturated and JobConsecutiveFailures in particular are workload-dependent — tune them to your fleet size and job mix.

Alert	Expression	For	Severity	Why this threshold
QueueDepthSaturated	kici_orch_dispatch_queue_depth{status="pending"} > 50	10m	warning	Pending work should drain as agents pick it up. A sustained backlog means the scaler is starved or stuck. 50 is deployment-size-dependent — tune to your fleet.
AgentsAllGoneWithBacklog	(sum(kici_orch_agents_active) == 0) and (sum(kici_orch_dispatch_queue_depth{status="pending"}) > 0)	10m	critical	Zero agents while jobs are queued is a hard stall. It also catches a lost connection to the hosted Platform: if the relay drops and the fleet drains, queued work has nothing to run it.
StaleRunsPresent	kici_orch_stale_runs_current > 0	15m	warning	Any stale run signals an agent that stopped heartbeating. The 15-minute window avoids flapping on transient detection.
EventDLQGrowing	increase(kici_orch_event_dlq_total[1h]) > 0	0m	warning	DLQ rows are never cleaned by TTL, so growth — not standing depth — is the pager. Any new admission means an event exhausted its retries.
EventDLQBacklog	kici_orch_event_dlq_depth > 0	30m	info	Standing DLQ depth is a todo, not a page. Informational so you triage the backlog without being paged for it.
ScalerSpawnFailing	increase(kici_orch_scaler_spawn_failures_total[15m]) > 0	0m	warning	Any spawn failure means some jobs cannot get an agent — worth a look at the backend and resource caps.
JobConsecutiveFailures	kici_orch_job_consecutive_failures > 3	0m	warning	A scheduled job failing three-plus times in a row is stuck, not a blip. Tune the count to your job mix.
DeclaredHostUnreachable	kici_orch_declared_hosts_unreachable > 0	10m	warning	A statically-declared roster host that is unreachable cannot take work.
DbCollationDrift	kici_orch_db_collation_drift > 0	0m	warning	Collation drift can silently hide present rows from text-index lookups — reindex after aligning the collation version.
StateReplayBreakerOpen	increase(kici_orch_state_replay_breaker_trips_total[15m]) > 0	0m	warning	The orchestrator gave up replaying run state to the Platform after repeated rejections and connected without it, so the Platform run mirror is knowingly stale. The 15m increase keeps the alert true long enough to survive Alertmanager grouping.
ScalerReapBlocked	kici_orch_scaler_reap_blocked > 0	15m	warning	The event-scaler teardown backstop refuses to act while this coordinator is connected to none of its known peers, because a partition and a dead peer are indistinguishable without a quorum. While it is off, a provision whose agent never registered is never torn down. On a 3+ node cluster a minority node reads 1 while the majority still reaps, so scope this to the leader or expect one series per minority instance. Firing on two coordinators at once — or on one whose peer your monitoring also cannot reach — means a partition, not a dead node. While it lasts, neither restart a coordinator nor add a third: both re-arm the reachable side’s sweep against the isolated side’s live agents.
ScalerProvisionsStuckUnseen	kici_orch_scaler_reap_unseen_provisions > 0	45m	warning	Provisions whose agent has been registered nowhere for longer than the flap grace, being timed before teardown. An agent absent only briefly — a reconnect, a peer flap — never reaches the gauge. The clock is leader-local, so leader churn faster than --scaler-reap-stranded-timeout-ms restarts it forever and the provision keeps billing. 45m sits above the 30m default window so a normally-expiring absence never fires — raise it if you tuned that knob up.
ScalerCapCheckUnreachable	increase(kici_orch_scaler_cap_lock_failures_total{reason="unreachable"}[10m]) > 0	15m	warning	An event scaler counts maxAgents across the whole cluster, so every spawn first takes a lock in the orchestrator database. When that check cannot run the spawn is refused rather than admitted — safe, but event-scaler provisioning stops cluster-wide while it lasts. Jobs stay queued and are re-offered once it recovers. Only reason="unreachable" is alerted; reason="contended" is a healthy database under load and belongs on the capacity dashboard. The 10m window keeps the condition continuous on an orchestrator that scales every few minutes, and 15m outlives the ~11m a single isolated burst stays true, so a blip or a rolling database restart cannot reach it.
ExternalProvisioningFailing	sum by (scaler) (increase(kici_orch_scaler_external_provision_timeout_total[1h])) >= 2	15m	warning	An event scaler hands provisioning to your workflow, which drives your provider, so the orchestrator never sees the provider’s error. The one signal it does have is generic — the scale-up went out and no agent came back — and it covers a provider incident, an exhausted quota, and a broken boot script the same way. The window is coupled to the orchestrator’s own backoff rather than chosen for its own sake: a scaler that keeps failing is deferred between attempts by scaler_provision_backoff_max_ms (15 min by default), so at steady state it times out about once every 20 minutes. A short rate window is empty for most of that cadence and would leave the rule dark through the exact outage it is written for; an hour always spans at least two attempts. The threshold is >= 2 because increase() extrapolates to the window edges. A counter that is already non-zero when the window opens reports about 1.01 for a single step, so > 1 would page on one unlucky spawn. Widen the window if you raise that setting past ~25 minutes.
PgListenChannelDisconnected	kici_orch_pg_listen_connected == 0	5m	critical	The orchestrator subscribes to four PostgreSQL LISTEN channels. Losing one is silent: the process stays up and /health and /ready stay 200 while nothing published on that channel arrives, so an event never routes, a source added in the dashboard is never loaded, a generic webhook source change never registers, or a dashboard write-policy change never takes effect. The listener reconnects on its own with backoff capped at 30 seconds, so a database failover or restart clears in seconds and never reaches this rule. Five minutes at zero means reconnection is failing rather than in progress — check that the database is reachable and that the connection pool is not exhausted.

Watching Platform connectivity

The orchestrator relays webhooks and dispatch over a connection to the hosted KiCI Platform. Two pack signals reflect its health without a dedicated connectivity metric: the Webhooks received rate panel shows inbound flow drying up, and AgentsAllGoneWithBacklog fires when the fleet drains while work is still queued — the shape a dropped relay produces once existing agents finish.

See also

• Monitoring & tracing