Applies to: clustered deployments (multiple instances sharing a config source, lease store, and outbox/DLQ stores). Audience: operators running a config change across a fleet. Risk: high — some live changes strand durable records or split lease ownership if rolled instead of drained.
Current behaviour: a clustered live reload is rejected fail-closed. Every non-no-op live reload of (or into) a clustered deployment is refused by the runtime and by the AWS composition root, keeping the running instance on its last-good config. Clustered config changes MUST go through the externally coordinated whole-cohort procedure in Cluster Config Rollout (stage → validate all → quiesce → drain/stop all → commit → start all → verify version/readiness barrier → re-enable ingress, with whole-cohort rollback). A genuine no-op re-emit is still accepted.
Historical hazard this guard now prevents. Reconfiguration is per-process: each instance watches its own config source, reloads, validates, and swaps its runtime independently, with no cluster-wide version barrier and no coordinated rollback (Scenario 10). Before this guard, rolling a config change through a cohort let the fleet run a mix of old and new definitions until every instance converged — indefinitely if one stayed wedged on a config it could not load. That split-version window is exactly what the fail-closed guard now blocks: local config CAS / reference tracking is not cluster consensus and never was a version barrier, so it must not be relied on for live cohort reconfiguration. The classification below describes single-process (standalone) reload behaviour and the reasons each change is unsafe to roll across a cohort.
Safe to roll live in a standalone (single-process) deployment (eventually consistent across a fleet if you were ever to run one, but in a clustered deployment even these are refused live — replace the cohort per the runbook above): routing, policy, and transformation changes. The same message class may be handled under the old or new definition during a divergence window, but no records are stranded and no lease is split.
Disallowed live / rolling — these are cluster invariants. Changing any of them under a rolling reload splits ownership or strands durable records:
session_id on a lease-bearing exclusive session — changes
the lease identity, so two instances can hold “its” lease in different keys
and drain independently (duplicate sends + stranded backlog).type or backing
path/table — repointing a store live strands durable records/history.shared_outbox partition, or
removing/renaming a persistent/exclusive MQTT session identity.These are hard-refused at swap time, per process, not merely warned. The
bridge rejects the reload and keeps the OLD runtime serving (metric
ConfigReloads{state="failure"}):
type/path/table change.session_id change.The only override discards the stranded backlog by design — do not set it for a routine
change. Separately, config validation rejects clustered-invalid shapes (for
example cluster endpoint and clustered exclusive HTTP direct_hold rules) at load, before any swap.
Roll a cluster-invariant change with a full stop/restart, never a rolling reload:
OutboxDepth to reach zero on every instance
(monitoring.md#key-metrics) —
confirm no OutboxStranded and no pending records before proceeding
(Outbox backlog / stuck drain).Persistent/exclusive MQTT filter removal has a stricter protocol because a broker may pin an unacknowledged shared delivery to the old ClientID. Preserve the managed-filter ledger and follow the persistent MQTT managed-filter migration runbook; a terminal migration-required result means restore the exact old identity and handlers, drain, then retry. The destructive-reload override does not make this safe, and GoBridge does not claim portable broker redistribution.
version CAS field guards concurrent commits to a shared file; it does not gate
the per-instance apply and is not a cluster rollout or rollback barrier.Scrape the running config version from every instance and confirm they agree:
for host in $FLEET; do
curl -s -H "X-API-Key: $MONITOR_API_KEY" \
"https://$host/api/v1/monitor/topology" | jq '{host: "'"$host"'", config_version, running}'
done
GET /api/v1/monitor/topology exposes config_version when a config provider is
wired alongside running. Every instance should
report the same config_version with running: true. Treat persistent version
divergence as an alertable condition; if an instance lags, it is wedged on a
config it cannot load — inspect its logs before forcing anything.