gobridge

AWS Deployment Topologies

Overview

The three shipped topologies — single task, replicated cluster, and the DynamoDB-coordinated HA profile — and what each one guarantees. The HA sections cover the rules that make failover safe: stable identities, least-privilege task roles, honest alarms, and the credentialed proof that a standby really took over.

Part of the AWS Deployment Overview.

Config Source by Topology

Topology selects how tasks coordinate; Bootstrap.ConfigSource separately selects where they read and write the bridge document.

Facade file dynamodb EFS
GoBridgeSingle Default Supported Only for file config or SQLite store paths
GoBridgeCluster Required Rejected at synth and runtime validation Required
GoBridgeDynamoDBHA Default Supported None with DynamoDB config and DynamoDB stores

Empty config_source means file in every topology. With either source, only control may initialize or update config; workers are read-only. DynamoDB config uses one shared CAS-versioned item in a facade-owned table, separate from the HA data and rollout tables. Switching the source does not migrate the old document. See configuration storage.

Replicated and Coordinated Deployments

The CDK library deliberately exposes two different multi-task profiles:

Facade Coordination model Intended use Failover objective
GoBridgeCluster filesystem_replicated; independent replicas read one EFS config Scale independent routes horizontally None. It has no active/standby takeover and no coordinated failover SLO.
GoBridgeDynamoDBHA dynamodb_coordinated_ha; one lease holder plus warm standbys Exclusive MQTT continuity with shared-outbox fencing Explicit per-route failover_slo, admitted by the builder and measured externally.

GoBridgeCluster remains unchanged. It rejects route.session and shared_outbox; it must not be described as HA. GoBridgeDynamoDBHA deploys one config-control task and at least two worker tasks across a subnet selection that spans at least two Availability Zones. All three tasks participate in DynamoDB lease acquisition, so the normal steady state has one active holder and at least two warm candidates. Every service uses a 0/100 non-overlapping replacement with Availability Zone rebalancing disabled, and the worker services are deployed after the control service. Only control may initialize absent configuration; workers wait idle for a valid document without any seeder dependency.

0/100 caps total tasks at the desired count, so a second cohort never runs beside the first. It constrains counts, not ORDER: on the autoscaled shape, at a desired count of two or more, the ECS scheduler may still replace in batches, so a revision that changes durable session identity or store targets still needs the scale-to-zero procedure in the cluster config rollout runbook. Expect an ingress gap and a breaching warm-standby alarm for the duration of an autoscaled deploy. The static member-slot shape does not have that gap: each slot is a single-task service and the slots are chained, so a deploy replaces one slot at a time and the rest of the cohort keeps serving — at the price of a deploy whose duration grows with the roster.

Two worker shapes

GoBridgeDynamoDBHA deploys its workers in one of two shapes, and the choice decides whether the cohort can take a live config change:

Shape Selected by Worker tasks Live coordinated rollout
Autoscaled workers (default) MemberSlots unset One ECS service, WorkerDesiredCount interchangeable tasks No. bridge.cluster.rollout: coordinated and a non-empty bridge.cluster.members are both rejected at synth.
Static member slots MemberSlots set One single-task ECS service per roster member, each with its own task definition and member_id Yes. The barrier runs; see the cluster guide.

The rejection is not a policy preference, it is the identity model. The rollout barrier freezes bridge.cluster.members as its membership epoch and counts acknowledgements against it, so a member must announce the same member_id after a restart. An autoscaled task gets a fresh ECS task id on every placement, so it can never re-enter the roster it left; a cohort of such tasks could never reach a quorum, and a half-satisfied cohort would commit generations no member applies. Rejecting the shape at synth beats deploying a stack that can only fail at boot.

With MemberSlots the construct additionally creates the retained, deletion- protected rollout coordination table (named <bridge.id>-rollouts from the shared config document, the same source as the three store table names), grants every task role exactly dynamodb:GetItem and dynamodb:PutItem on it — the only two calls the rollout store makes — stamps each slot’s member_id and the generation-zero baseline digest into that slot’s bootstrap document, and orders the control slot first and then each worker slot after the previous one, so a deploy replaces at most one slot at a time. WorkerDesiredCount is rejected alongside MemberSlots: the roster is the slot count, and scaling a slot past one task would run two processes under one member_id.

A config change that the barrier classifies as replacement-required — and every change to the deployment profile itself, including the image and the roster — is still a CloudFormation deploy in both shapes.

A deploy that changes a fingerprinted field needs the shared target document to change with it. Embedding new config in an image never overwrites an existing target. Use the scale-to-zero procedure in the cluster config rollout runbook: validate the exact config, quiesce and stop the cohort, write the target, then start the replacement cohort and verify convergence before restoring intake. Otherwise members reject the target because its deployment fingerprint differs from bootstrap. Existing admin changes survive ordinary control restarts.

With valid bootstrap settings, missing config leaves the control plane live but not ready and the data plane idle. After clustered activation, confirmed absence stops intake and signals process exit and replacement, not same-process idle. Standalone operation can drain to idle; uncertain teardown also exits. Source read failures retain the last successful config as degraded. A warm standby counts as activated without needing Full readiness. See initialization lifecycle.

Coordinated HA data plane

GoBridgeDynamoDBHA creates three encrypted, point-in-time-recoverable, delete-protected, retained PAY_PER_REQUEST data tables, plus a rollout table with MemberSlots. Selecting config_source: dynamodb adds a separate retained config table; it is not part of the data-table count below. The key/index shapes are the adapter contracts, not deployment inventions:

Table Schema TTL invariant
Lease (gobridge-leases default) PK string hash key; no sort key or indexes Disabled. The row carries the permanent monotonic fencing version. Deleting it can reset fencing and permit split brain.
Shared outbox (gobridge-outbox default) PK/SK; ExpiryIndex KEYS_ONLY, RecordIDIndex KEYS_ONLY, ClaimIndex ALL Enabled on ttl only for terminal records and old fence metadata. Pending work is never TTL-reaped.
Managed subscriptions (gobridge-managed-subscriptions default) storage_identity string hash key Disabled; exact MQTT filter history is durable.
Rollout coordination (<bridge.id>-rollouts, only with MemberSlots) PK string hash key; no sort key or indexes – the rollout aggregate is one row Disabled. The row holds the cohort’s last committed config artifact, the point every restarting member recovers to.

The data API is DynamoDBHAData, returned by bridge.Data(). It exposes the lease, outbox, and managed-subscription table objects, names, and ARNs, plus the rollout coordination table when MemberSlots is configured. RolloutTable(), RolloutTableName(), and RolloutTableARN() return nil when no rollout table is provisioned.

On-demand billing is appropriate for bursty takeover and outage recovery, but it does not eliminate hot keys. A single Exclusive MQTT session concentrates the outbox on one SESSION#... partition. Split unrelated workloads across session IDs before that partition approaches DynamoDB limits. Preserve ClaimIndex to avoid the adapter O(backlog) compatibility scan, and monitor the sparse ExpiryIndex guidance for expiry-heavy traffic.

Identity and endpoint rules

The shared bridge YAML must use deployment_mode: clustered, DynamoDB lease, outbox, and managed-subscription stores, delivery_mode: shared_outbox, ack_after: outbox_persist, and explicit failover_slo plus startup_allowance. Every Exclusive MQTT standby uses the same broker domain, client_id, clean-start/session-expiry behavior, and managed-subscription storage identity. client_id_suffix is rejected for Exclusive sessions because a per-task MQTT identity strands queued broker state after holder loss.

The facade also stamps two identities plus the exact table names into deployment-owned bootstrap, and every process validates the selected-source logical config against them before store or transport planning. An existing document cannot bypass synth-time admission:

Static bridge.cluster.endpoints are rejected by this profile. The bootstrap registers the existing ECS metadata endpoint resolver and each holder writes its own reachable endpoint into the lease row. This endpoint also lets the credentialed proof map the lease to one exact ECS task without guessing.

Least-privilege task roles

Either task role can become active, so control and workers receive the same narrow data access:

No task role receives DynamoDB table creation/update/deletion, UpdateTimeToLive, wildcard actions, or wildcard index resources. The external proof principal receives no grant from the facade; its operator policy must separately allow the required ECS/DynamoDB reads and cloudwatch:PutMetricData/metric query calls.

Alarms and objective honesty

The HA form of GoBridgeAlarms covers running/desired task count, minimum warm standby, all-table DynamoDB throttles/system errors, lease expiry and takeover flapping, shared-outbox depth/drain latency/failures, DLQ signals, and FailureToFullDuration.

FailureToFullDuration is emitted by the credentialed external health/failover probe, not by the runtime. The probe conservatively starts timing before the verified holder StopTask request, waits for that exact task to be STOPPED, requires both lease owner and fencing version to change, and waits for a different exact successor to report ServiceLevelFull. A sample is classified warm only when that successor task ARN was already running in the pre-failure standby snapshot; a replacement winner is classified cold. It publishes one no-dimension millisecond sample in the configured deployment namespace. The alarm uses TreatMissingData=NOT_BREACHING; the release test immediately queries CloudWatch for the exact sample and fails if it is absent. Continuous SLO evidence therefore requires an operator-scheduled external probe.

The checked example/fixture objective is 120 seconds. Admission proves only that configured worst-case terms fit that ceiling. It does not prove an achieved production percentile. No 30–60 second claim is made. Publish a tighter target only after enough warm and cold samples from the actual image, VPC, broker, credentials, and AWS account support it. OutboxDrainLatency is a drain-cycle measurement, not direct oldest-record age; inspect the oldest pending item when triaging backlog age.

Credentialed failover proof

The test runner needs Docker, CDK deploy/destroy credentials, a two-AZ VPC, a reachable TLS MQTT broker, existing SecureString admin/MQTT parameters, CloudWatch metric write/read permission, and VPC routing to task private addresses. The fixture opens monitor port 8081 only from GOBRIDGE_INT_HA_PROBE_CIDR; production security groups remain unchanged.

Required variables:

GOBRIDGE_INT_HA=1
GOBRIDGE_INT_AWS_ACCOUNT
GOBRIDGE_INT_AWS_REGION
GOBRIDGE_INT_VPC_ID
GOBRIDGE_INT_AVAILABILITY_ZONES
GOBRIDGE_INT_SUBNET_IDS
GOBRIDGE_INT_PUBLIC_SUBNET_IDS
GOBRIDGE_INT_VERSION
GOBRIDGE_INT_HA_MQTT_BROKER_URL
GOBRIDGE_INT_HA_MQTT_CLIENT_ID
GOBRIDGE_INT_HA_MQTT_CREDENTIAL_PARAM
GOBRIDGE_INT_HA_ADMIN_PARAM
GOBRIDGE_INT_HA_PROBE_CIDR

GOBRIDGE_INT_VERSION must name a published AWS profile lib module version that supports embedded config and -initial-config-digest. No released train has published lib yet, so this needs a train cut after it joined. Each credentialed fixture uses ImageFromGoBuild to build an image containing that fixture’s parsed config. The harness rejects GOBRIDGE_INT_IMAGE; unset it rather than passing a registry reference — an existing registry image cannot substitute for the versioned build.

The availability-zone, private-subnet, and public-subnet lists must have the same order and cardinality. The harness imports these concrete attributes and produces an assembly with no VPC lookup context.

Optional GOBRIDGE_INT_HA_SAMPLES controls separate warm/cold sample counts (1–20, default 1). Run:

cd deployment/aws/cdk
GOBRIDGE_INT_HA=1 go test -count=1 -v -tags=integration_aws -run TestHA_FailoverStopsVerifiedLeaseholder ./integration

When GOBRIDGE_INT_HA=1, missing variables, credentials, outputs, network reachability, owner/fence changes, Full readiness, or the exact CloudWatch sample fail the test. Without that explicit request, the credentialed build-tag test is skipped and no AWS deployment occurs.

Retain the built image digest with the module version and failover evidence. This fixture image contains its own initial document; it is not the generic published runtime image.

This site is open source. Improve this page.