Operational Runbooks
Symptom-first, incident-shaped runbooks: each starts from what you see, moves to
diagnosis, then to action. Every step is grounded in existing behavior — real
metric names (monitoring), error codes
(troubleshooting), and endpoints
(HTTP API).
Incident runbooks
| Runbook |
Start here when |
| Broker outage / reconnect storm |
Sessions reconnect in a loop, throughput drops, CONNECTION_LOST / BROKER_BUSY in logs. |
| Poison message / DLQ growth |
DLQEntries climbing, the same envelope fails repeatedly, POISON_MESSAGE. |
| Lease flapping / split-brain |
Leadership bounces between instances, LeaseTransfers climbing, duplicate deliveries suspected. |
| Credential expiry / rotation failure |
Auth fails after a rotation, CredentialResolveFailure / CredentialStaleServed non-zero. |
| DynamoDB store outage / throttling |
Lease/outbox/DLQ store slow or erroring, THROTTLED, LeaseRenewLatency / OutboxDepth climbing, DLQWriteFailures. |
| Outbox backlog / stuck drain |
OutboxDepth / OutboxDeferred rising, OutboxDrainStalled non-zero, DrainSkippedNoLease climbing. |
| Node down / failover |
An instance/task died — confirm the standby took over. LeaseTransfers / LeaseExpiries advanced. |
| Config rollback |
A committed config change caused errors and must be reverted. |
| MQTT ingress poison |
MQTTIngressPoisonDropped non-zero — an authorized publisher sends packets the bridge is configured to refuse; each drop is acknowledged loss. |
| Stuck MQTT settlement |
MQTTOldestUnsettledAge climbing and MQTTReceiveWindowUtilization near 1.0 — ingress slows or stops while the session stays connected and no route reports an error. |
| MQTT SUBACK rejection / QoS downgrade flap |
ReconcileFailures climbs every ~30s and never converges; readiness stuck below Full; exclusive session churns its lease. |
| Orchestrator kill before shutdown drain |
The task exited 137 (SIGKILL) or 2 before graceful drain finished; check for best-effort loss and redelivery duplicates. |
Procedures
| Runbook |
Purpose |
| Image upgrade / rollback and SQLite durability |
Image version rollout and rollback; SQLite backup/restore and the durable-volume requirement. |
| Cluster reconfiguration |
Roll a config change across a fleet: allowed vs. disallowed live changes, drain-and-stop, convergence verification. |
| Cluster config rollout (whole-cohort replacement) |
The manual stop-and-restart for a change that can’t roll live — a replacement-required change, or any change in a non-coordinated / file-sourced cohort: stage, validate all, quiesce, drain/stop all, commit, start all, verify version/readiness barrier, whole-cohort rollback. Coordinated cohorts roll live-safe changes with no downtime — see the cluster docs. |
| Persistent MQTT managed-filter migration |
Remove durable wildcard/shared filters safely; restore, drain, and retry when a broker pins an unacknowledged shared delivery. |
| External config writers must write atomically |
An external tool writes the watched config file; ensure temp-file + rename, never truncate-in-place. |
| DynamoDB outbox table schema |
Required outbox table/index shape, why ClaimIndex must be Projection: ALL, and how to read a preflight rejection. |