gobridge

Design: Coordinated cluster config rollout (barrier protocol)

Overview

Status: fully implemented — every slice has shipped. This is the canonical protocol design. The shipped decisions are recorded authoritatively in ADR 0013 (base barrier) and ADR 0014 (confirm window); where this doc and an ADR ever disagree, the ADR wins. §8.1 (the confirm window) is the final piece of the protocol — there is nothing beyond it.

The barrier is wired into both runtime hosts — bridge.Supervisor and the shipped file-based bootstrap.App — through a bridge.ClusterRolloutDriver bound to a ports.RolloutHost seam (the host is a port, not a bridge type), so a coordinated cohort performs live-safe rollouts and a non-coordinated one keeps the ADR 0012 refusal.

Companion docs in this folder: cluster-config-rollout-confirm-window.md (confirm-window code-level detail) and cluster-config-rollout-research.md (prior art & split-brain analysis — the external specs this design reuses).

Date: 2026-07-26 · Relates to: ADR 0012, ADR 0013, ADR 0014 · Operations: docs/runbooks/cluster-config-rollout.md · Plain-language configuration guide: docs/cluster/README.md

1. Problem

Each GoBridge process watches, validates, and applies configuration independently. There is no cluster-wide version barrier, no all-member readiness gate, and no coordinated rollback — so the runtime refuses every non-no-op live reload of (or into) a clustered deployment, fail-closed (ADR 0012, guard in bridge.Supervisor.apply and bootstrap.App.applyLogicalConfig). Operators must replace the whole cohort.

Goal: allow an operator to POST a config change to any one node and have the cluster decide on it atomically: every member stages it as a candidate, and only when all members acknowledge does it commit; any failure aborts everywhere and the old config keeps serving. Applying the committed generation is then per-member and eventual — see G2/G2b below and ADR 0013.

2. Goals and non-goals

Goals

Non-goals

3. Protocol overview

flowchart TB
    Operator[Operator stages config] --> Proposal[Propose generation and membership epoch]
    Proposal --> Members[Each member validates and builds candidate]
    Members --> Votes[Ack or Nack]
    Votes --> Coordinator[Fenced coordinator reads votes]
    Coordinator -->|Every member acknowledged| Commit[Commit generation]
    Coordinator -->|Nack, timeout, or epoch change| Abort[Abort generation]
    Commit --> Apply[Each member applies and checks convergence]
    Abort --> Discard[Each member discards candidate]

States: Proposed → Staging → Committed | Aborted (terminal). One active rollout at a time; generation is monotonic. The confirm window (§8.1) adds two further terminal states, Confirmed | Reverted, reached only when a window is set.

4. Domain model (Layer 1)

Placement: domain/persistence — the rollout barrier is the same family as leases and outbox claims (coordination-through-durable-state, fencing, TTL); it reuses persistence.LeaseToken for fencing. A separate domain/coordination context was considered and rejected: it would hold two value types and force a new arch component + lint mapping for no added invariant clarity.

New value types (no behavior beyond invariant checks):

type RolloutState string // Proposed | Staging | Committed | Aborted

type Rollout struct {
    Generation      uint64        // monotonic, one active at a time
    State           RolloutState
    ConfigDigest    string        // sha256 of canonical candidate config
    ConfigVersion   int           // the BridgeConfig.Version being rolled out
    MembershipEpoch []string      // sorted member IDs frozen at Propose
    Acks            map[string]RolloutAck
    Reason          string        // abort reason / nack aggregation
    Deadline        time.Time     // coordinator aborts after this
}

type RolloutAck struct { MemberID string; BuildDigest string; At time.Time }

type RolloutProposal struct {
    ProposerID    string
    ConfigDigest  string
    ConfigVersion int
    Members       []string      // epoch snapshot
    TTL           time.Duration // rollout deadline budget
}

Invariants (enforced by domain constructors + store conditions):

The confirm window (§8.1) adds I6 (a member converges at most once, only under an active window) and I7 (Confirm requires the whole epoch converged).

5. Ports (Layer 2 boundary)

One new small port (hexagonal; interface stays minimal per project rule):

// ClusterRolloutStore coordinates a staged, all-member config rollout
// through durable conditional writes. Implementations must make Propose,
// Commit, and Abort atomic check-and-set operations.
type ClusterRolloutStore interface {
    Propose(ctx context.Context, p persistence.RolloutProposal) (persistence.Rollout, error)
    Ack(ctx context.Context, gen uint64, memberID, buildDigest string) error
    Nack(ctx context.Context, gen uint64, memberID, reason string) error
    Commit(ctx context.Context, gen uint64, tok persistence.LeaseToken) error
    Abort(ctx context.Context, gen uint64, tok persistence.LeaseToken, reason string) error
    Current(ctx context.Context) (persistence.Rollout, error)
    // confirm window (§8.1): Converge, Confirm, Revert
}

Reused, unchanged:

Adapters (Layer 3): adapter_store_dynamodb_rollout (DynamoDB, conditional writes — same idioms as dynamodblease), adapter_store_memory_rollout (tests / single-binary). Both register per PLUGIN.md; both must pass the ports/storetest.RunClusterRolloutStoreTests conformance suite.

Store invariants (split-brain prevention by construction — research §3): coordination tables (rollout, lease) are single-Region (multi-AZ tolerance is inherent: DynamoDB replicates per-partition across AZs with internal Paxos). They must never become MREC global tables — cross-Region last-writer-wins would let two “successful” conditional writes diverge, reintroducing split-brain at the store. All rollout decision reads use ConsistentRead; never GSIs/Streams. Under these invariants, divergent histories needing “merge” cannot exist: concurrent proposals resolve to one CAS winner and one ConditionalCheckFailedException.

6. Orchestration (Layer 2, bridge/)

Proposer (any node, from the existing admin config-transaction API): commit of a txn in coordinated-cluster mode writes the candidate config to the config source (inactive), computes the digest, and calls Propose.

Applier (every node, in the Supervisor): on observing Proposed/Staging with an unseen generation — run rollout-class preflight (§8); fetch + digest- verify candidate; validate; build a candidate runtime via the existing applyPrepareCommit prepare path but do not swap; Ack (or Nack with the error). Then wait: on Committed → complete the prepared swap (post-swap the convergence watch runs as today); on Aborted → discard the candidate runtime (existing candidate-cleanup path, RECONFIG-2). Store notifications are hints, never truth: every decision re-reads the rollout row with ConsistentRead (research rule 11). Each node is itself a token-checking resource (research §3): it persists the highest (epoch, generation) it has applied and rejects anything lower — a paused-then-resumed coordinator’s late push is harmless at the node, not just at the store.

Coordinator (whichever node holds the rollout lease): poll Current + membership; Commit when acks cover the epoch; Abort on any Nack, on Deadline expiry, or on membership-epoch change. Coordinator work is idempotent and resumable — all state is in the store, so a successor elected after a crash continues from whatever it reads. A freshly elected coordinator waits out one full previous-lease duration before its first side effect (Chubby-style lock-delay — belt and braces over the fencing epoch; DynamoDB itself does this internally for its partition leaders).

Joiner rule: a starting member adopts only the last Committed configuration through the committed-artifact path; it never acks a rollout proposed before it joined (its ID is not in the epoch).

Guard change: the ADR 0012 refusal remains the default. It is lifted only when bridge.cluster.rollout: coordinated is configured and the delta is live-safe (§8); everything else still refuses fail-closed exactly as today.

7. Failure matrix

# Failure Outcome Mechanism
F1 Member crashes before Ack Rollout aborts at deadline; survivors keep old config; member rejoins on old (still-committed) gen Deadline + joiner rule
F2 Member Nacks (validation/build fails) Abort everywhere; candidates discarded Coordinator on first Nack
F3 Coordinator crashes mid-rollout Rollout lease expires (TTL); successor resumes from store state and commits/aborts LeaseStore election + idempotent coordinator
F4 Two concurrent proposes Second Propose fails the conditional create (I1) CAS
F5 Deposed coordinator tries Commit/Abort after the live one decided Rejected — stale fencing token (I3) Token condition
F5b Deposed coordinator decides first (no decision recorded yet) Not fenced — accepted. Fail-safe: a zombie Commit still needs the full ack barrier (I2), a zombie Abort just keeps the old config serving. Bounded in practice by the successor’s lock-delay (§6) Residual — accepted (§11)
F6 Membership changes mid-rollout (join/leave) Abort; operator retries (cheap — nothing swapped) Strict epoch equality; simplest safe rule
F7 Member crashes after Commit, before its swap Rejoins and boots the committed gen — same config, no split Joiner rule
F8 Committed config fails to converge on a node (e.g. broker unreachable) No distributed rollback; that node latches ConfigDegraded via the convergence watch, alarmed — parity with single-node behavior (unless a confirm window is set, §8.1) §2 N5
F9 Store unavailable mid-rollout No state flips possible; members keep old config; rollout resolves (or deadline-aborts) when the store returns All transitions are store writes
F10 Candidate bytes tampered / mismatched Member digest check fails → Nack → abort Digest in rollout row

8. Rollout classes and config surface

Per-node preflight classifies the delta before Ack:

New config keys (validated in config/validate.go):

bridge:
  deployment_mode: clustered
  cluster:
    rollout: refuse | coordinated   # default: refuse (today's behavior)
    members: [node-a, node-b]       # required when coordinated: the cohort roster

members is the membership epoch the barrier freezes (F6) — NOT cluster.endpoints, which is this instance’s capability map. coordinated also requires a versioned, CAS-capable config source and a wired rollout store; those are composition-root wiring, invisible to the blueprint validator, so a root that wires no rollout store makes every coordinated reload fail closed. Each process announces its own member_id (bootstrap config), which must appear in members and survive restarts.

8.1 Confirm-window extension (Model B — opt-in, ✅ implemented)

The base protocol’s Ack proves validated + built, not converged against the real broker (research §4, Model A vs B). An opt-in confirm window adds the NETCONF/NSO “provisional apply with deadman timer” layer on top (research rules 8 and 10):

bridge:
  cluster:
    rollout: coordinated
    confirm_window: 90s     # 0 (default) = base protocol only

Cost note (why this is opt-in): for exclusive-identity MQTT sessions a failed trial costs two service disruptions (apply + revert) — the JunOS trade, accepted when a bad config staying active is worse. See cluster-config-rollout-confirm-window.md for the code-level detail and the two implementation decisions (revert target = cached pre-swap config; confirm must land within the window).

9. Observability

10. Test plan (all gates deterministic — TESTS.md rules)

Unit (domain/persistence, bridge/):

Conformance (ports/storetest):

Integration (tests/integration/, ddblocal + mqttlocal):

Long-running (tests/longrunning/, nodeProcess multi-process harness — real processes, real DynamoDB, real broker; barriers are stdout tokens + store rows, never sleeps):

11. Implementation status, decisions & coverage

Every slice has shipped and lands green on make lint + make test. The base barrier is ADR 0013; the confirm window is ADR 0014. The decisions below are the ones the ADRs do not fully carry.

Membership authority (Q1). A dedicated static roster key bridge.cluster.members, plus a per-process member_id injected by the composition root (bridge.WithClusterRollout.MemberID). The roster is NOT cluster.endpoints: that key is this instance’s capability map ({http: …}), so a cohort keyed off it would freeze the epoch ["http"] — a one-member barrier, i.e. no barrier at all. The member id cannot be derived from bridge.instance_id (empty in a shared-config cohort so each task derives a unique runtime metric identity), so “this node is in its own roster” is admitted at startup, not by the blueprint validator.

Candidate transport (option b). Each member’s OWN config watcher delivers the candidate bytes; the rollout row carries only digest+version as the cross-member agreement check, and a booting member stages the document it booted so it can vote on a rollout that later carries it. Option (a) — pushing bytes through the row — was rejected as unnecessary machinery since every member already receives the change. Digest determinism holds because GoBridge performs NO per-node interpolation on the config load path (a shared.Secret is a literal carried in the document), so two members canonicalise the same document identically.

Durable last-committed artifact. Option (b) alone left a member with no durable answer to “what should I be running?” independent of the rollout row. Closed with a second coordination-store row (ROLLOUT#committed, ports.ClusterCommittedConfigStore) holding the round-trippable committed config BYTES + canonical digest (a ClusterRolloutConfig.Encode/Decode codec is injected because bridge may not import config/parser). The adopting member writes it (idempotent, monotonic); a joiner then boots on the committed config when current holds a candidate the barrier has not committed, and the applier reconciles to it when the active row moved on before a member observed the commit. It advances on Commit (base) and, under a window, only on Confirm (§8.1) — so a crash reboots onto the last confirmed generation. Without a deployment-admitted baseline identity, a candidate in current during the write→propose window cannot be distinguished from the baseline. The AWS HA profile supplies dynamodb_ha_baseline_config_digest to establish the generation-zero committed artifact. This is separate from creating an absent config-source document. Both file and DynamoDB baseline matching use bridge.DeploymentBaselineContentDigest, normalizing only the top-level version to zero because initialization assigns target version 1 independently of the embedded version. The committed artifact retains its actual stored version and full, version-sensitive bridge.ConfigArtifactDigest.

Initial configuration and deletion. Only control may initialize an absent source document, through strict CreateIfAbsent, at version 1. Existing documents always win. An embedded document never replaces an existing source or bypasses the rollout barrier. With valid bootstrap, no valid config means live control-plane services but an idle, not-ready data plane. After clustered activation, confirmed source absence stops new intake and signals process exit and replacement. A clustered runtime does not return to live idle, with or without the coordinated barrier. Uncertain teardown also exits. Do not continue processing the old or cached committed config after confirmed absence. Standalone runtimes may rebuild after safe quiescence; read errors instead retain the last successful config as degraded. First activation is independent of Full readiness; returning to idle does not rearm initialization in the same process. Watchers must preserve delete/recreate ordering and accept a recreated source version through the normal validation and rollout path. See initialization lifecycle.

Composition obligations. A coordinated root MUST wire config.Validate (an Ack proves the candidate passes the BlueprintValidator and builds, but not the runtime route-graph validation that runs at commit — without it a dangling reference is acked by all, committed, then fails every swap; G2 still holds — every member reached the same decision and the same outcome). It MUST also re-sync the config manager after a barrier swap (Manager.AdoptRunning / NotifyApplyResult), or ReconfigurePending / deep-health Degraded can latch despite correct convergence.

F5b residual — accepted. No coordinator-claim write before the first decision: every zombie outcome is fail-safe (a zombie Commit still needs the full ack barrier; a zombie Abort keeps the old config serving), and the coordinator renews its lease BEFORE every observation, so a deposed one steps down without deciding.

Q3 audit trail — overwrite, retaining the last. One rollout row (active or last-decided), readable via Current, /deephealth, and metrics. A ledger of N past rollouts is a store-schema change with no consumer yet.

Confirm window (§8.1). Two terminal states RolloutConfirmed/ RolloutReverted; a windowed Committed is non-terminal until a fenced Confirm (whole epoch converged) or Revert decides it. The coordinator checks the deadline BEFORE the confirm barrier, so a late-resuming coordinator reverts rather than flapping a cohort that already began to revert. The revert target is the cached pre-swap config (generation N−1), captured at the first provisional observation — always available, even for the first windowed rollout, which has no prior committed artifact.

Test coverage matches §10. Known harness limitation: the confirm-window deadman arm (one member never converges → whole cohort reverts) is proven deterministically in-process (TestClusterRolloutDriver_ConfirmWindow_DeadmanRevert, real coordinator + applier + store, injectable convergence). It is NOT reproduced multi-process — forcing one real member to swap-but-never-converge needs a controllable- readiness transport the nodeProcess harness lacks (the same limitation this design records for UC-CR6). When that transport lands, UC-CR9 gains the revert arm across real processes.

12. ADRs

The design decisions ship as ADRs: 0013 — Coordinated cluster config rollout (supersedes 0012 for live-safe deltas) and 0014 — Confirm window: provisional commit with deadman revert (extends 0013). Those are authoritative for shipped behavior; this section replaces the earlier draft ADR that was promoted verbatim at ship time.

13. Open questions — resolved, retained for provenance

Q Question State
Q1 Membership authority Decided: a dedicated static roster key bridge.cluster.members, plus an injected MemberID (§11). The earlier answer — the keys of cluster.endpoints — was wrong (that key is this instance’s capability map). Revisit when endpoint auto-discovery becomes a supported shape; gossip views would stay advisory either way — the frozen list in the rollout row is the safety input
Q2 Rollout deadline default Measured: N=3 propose→all-committed staging ≈ 1 s on DynamoDB Local; defaultRolloutTTL = 5 min retained with ample margin (NETCONF’s confirmed-commit default is 600 s)
Q3 Retain Aborted rollouts as an audit trail, or overwrite? Decided: overwrite, retaining the last. One row holds the active-or-last-decided rollout, exposed via Current, /deephealth, and the resolution counter. A ledger of N past rollouts is a store-schema change with no consumer yet
Q4 Strict all-ack (this design) vs Kafka-style “commit centrally, fence non-converged members out of serving” Deferred. Strict-abort/revert is simpler and matches the operator contract for cohorts ≤ ~10; the confirm window (§8.1) is the first thing that makes converged-vs-acked distinguishable. Revisit if cohorts grow beyond ~10
F5b Fence the first coordinator decision (claim write), or accept the residual? Decided: accept the residual (§11). Every zombie outcome is fail-safe, and the coordinator renews its lease before each observation
This site is open source. Improve this page.