Go live with alarms, dashboards, auto-scaling, hardened security, and full operational visibility. This scenario builds on Scenario 1 and adds everything you need for a production deployment that your on-call team can operate with confidence.
The consumer owns the registry image and its embedded initial document. CDK declarations do not overwrite existing config. Only control may initialize an absent target; workers stay read-only. See initial configuration.
You are deploying GoBridge to handle business-critical messages with defined SLA requirements. Operators need dashboards, alarms, and structured logs to diagnose issues without SSH access. Specific requirements:
flowchart TB
subgraph VPC ["VPC (Multi-AZ)"]
subgraph AZ1 ["AZ 1"]
T1[Fargate Task 1]
end
subgraph AZ2 ["AZ 2"]
T2[Fargate Task 2]
end
EFS[(EFS Config)]
--- EFS
--- EFS
subgraph Endpoints ["VPC Endpoints"]
VPCE[SSM / SQS / ECR / CW Logs]
end
end
T1 & T2 --> CWL[CloudWatch Logs]
T1 & T2 --> CWM[CloudWatch Metrics]
T1 & T2 --> XRay[X-Ray via ADOT]
CWM --> Alarms --> SNS[SNS Topic] --> PD[PagerDuty / Slack]
CWM --> Dashboard[CloudWatch Dashboard]
The GoBridge Dockerfile runs as user ID 65532. Enforce a read-only root filesystem
in the container definition by setting ReadonlyRootFilesystem: jsii.Bool(true).
The EFS config volume is writable for control and read-only for workers.
Do not make the control config mount read-only if it must create or update config.
kmsKey := awskms.NewKey(stack, jsii.String("Key"), &awskms.KeyProps{
Description: jsii.String("GoBridge SSM encryption"), EnableKeyRotation: jsii.Bool(true),
})
Note: CDK does not natively create
SecureStringparameters. Useaws ssm put-parameter --type SecureString --key-id <key-id>in your bootstrap script, then reference the ARN in the task role policy.
Grant only what the task needs – SSM reads by path prefix and KMS decrypt:
taskRole.AddToPrincipalPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{jsii.String("ssm:GetParameter")},
Resources: &[]*string{jsii.String("arn:aws:ssm:*:*:parameter/gobridge/prod/*")},
}))
taskRole.AddToPrincipalPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{jsii.String("kms:Decrypt")},
Resources: &[]*string{kmsKey.KeyArn()},
}))
The GoBridge facade (single or cluster) automatically grants
elasticfilesystem:ClientMount on the EFS filesystem to both task roles.
Only control receives elasticfilesystem:ClientWrite.
Eliminate NAT Gateway costs by creating interface endpoints for every AWS service GoBridge calls:
for _, ep := range []struct {
ID string
Svc awsec2.InterfaceVpcEndpointAwsService
}{
{"SSM", awsec2.InterfaceVpcEndpointAwsService_SSM()},
{"SQS", awsec2.InterfaceVpcEndpointAwsService_SQS()},
{"ECR", awsec2.InterfaceVpcEndpointAwsService_ECR()},
{"ECRDocker", awsec2.InterfaceVpcEndpointAwsService_ECR_DOCKER()},
{"CWLogs", awsec2.InterfaceVpcEndpointAwsService_CLOUDWATCH_LOGS()},
{"CWMetrics", awsec2.InterfaceVpcEndpointAwsService_CLOUDWATCH()},
} {
vpc.AddInterfaceEndpoint(jsii.String(ep.ID),
&awsec2.InterfaceVpcEndpointOptions{
Service: ep.Svc, PrivateDnsEnabled: jsii.Bool(true),
},
)
}
vpc.AddGatewayEndpoint(jsii.String("S3"), &awsec2.GatewayVpcEndpointOptions{
Service: awsec2.GatewayVpcEndpointAwsService_S3(),
})
Each interface endpoint costs ~$7.30/month. The S3 gateway endpoint is free.
Autoscaling is a property of the cluster facade and applies to the worker
service only (the control task is always a single EFS RW writer, DesiredCount
hard-coded to 1). It is opt-in: pass AutoScaling on ClusterProps. When
TargetCPU is 0 it defaults to 70. The single facade has no autoscaling.
workers := float64(2)
bridge := gobridge.NewCluster(stack, "Bridge",
&gobridge.ClusterProps{
WorkerDesiredCount: &workers,
AutoScaling: &gobridge.AutoScaling{
Min: 2,
Max: 8,
TargetCPU: 70,
},
// ... other props
},
)
| Behavior | Detail |
|---|---|
| Scaling target | Worker service average ECS CPU utilization at TargetCPU (70%) |
| Min workers | AutoScaling.Min (2) – floor on the worker DesiredCount |
| Max workers | AutoScaling.Max (8) – ceiling on the worker DesiredCount |
| Control task | Always 1 (not autoscaled) |
These alarms cover the failure modes that matter most. Each fires to an SNS topic that routes to your incident management system.
| Alarm | Metric | Threshold | Period | Severity |
|---|---|---|---|---|
| Unhealthy Tasks | ECS CPUUtilization SampleCount |
< 2 | 1 min | Critical |
| High Error Rate | RouteErrors / MessagesReceived * 100 |
> 5% | 5 min | High |
| CPU Utilization | ECS CPUUtilization |
> 80% | 5 min | Warn |
| DLQ Arrivals | DLQEntries Sum |
> 0 | 5 min | High |
| DLQ Depth | DLQDepth Maximum |
> 0 | 5 min | Warn |
| Message Loss | MessagesDropped Sum |
> 0 | 5 min | Critical |
| Config Reload Failure | ConfigReloadFailures (log metric) |
> 0 | 5 min | High |
DLQEntries is an INGRESS counter (arrivals, only ever increases), so its Sum
answers “did anything land in the DLQ this window”. DLQDepth is the standing
BACKLOG gauge (see monitoring) — it stays
lit while entries remain outstanding even after arrivals stop, so alarm on its
Maximum. MessagesDropped is the silent-loss counter: any terminal drop
settled without a DLQ record, so a non-zero Sum is real message loss. All three
ship in DefaultAlarms/DefaultRollupMetrics.
Example – DLQ alarm with SNS action:
dlqAlarm := awscloudwatch.NewAlarm(stack, jsii.String("DLQDepth"),
&awscloudwatch.AlarmProps{
AlarmName: jsii.String("GoBridge-DLQ-NonEmpty"),
Metric: awscloudwatch.NewMetric(&awscloudwatch.MetricProps{
Namespace: jsii.String("GoBridge/Runtime"), MetricName: jsii.String("DLQEntries"),
Statistic: jsii.String("Sum"), Period: awscdk.Duration_Minutes(jsii.Number(5)),
}),
Threshold: jsii.Number(0), EvaluationPeriods: jsii.Number(1),
ComparisonOperator: awscloudwatch.ComparisonOperator_GREATER_THAN_THRESHOLD,
TreatMissingData: awscloudwatch.TreatMissingData_NOT_BREACHING,
},
)
dlqAlarm.AddAlarmAction(alarmAction)
dlqAlarm.AddOkAction(alarmAction)
See the Monitoring Guide for the complete alarm definitions including the math-expression error-rate alarm.
The dashboard gives the on-call team a single pane of glass.
| Row | Widget | Metric | Type |
|---|---|---|---|
| 1 | Throughput | MessagesReceived, MessagesSent |
Line graph |
| 1 | Delivery Latency | DeliveryE2ELatency p50, p99 |
Line graph |
| 2 | Error Rate | RouteErrors / MessagesReceived * 100 |
Single value |
| 2 | ECS CPU & Memory | CPUUtilization, MemoryUtilization |
Stacked area |
See the complete stack below for the full CDK dashboard code. The Monitoring Guide covers the dashboard JSON layout in detail.
Organize parameters under a path prefix for clean IAM scoping:
/gobridge/prod/admin-api-key (SecureString)
/gobridge/prod/monitor-api-key (SecureString)
/gobridge/prod/mqtt-password (SecureString)
Reference them in the bootstrap config:
Bootstrap: gobridge.Bootstrap{
BridgeID: "gobridge-prod", ConfigFilePath: "/var/lib/gobridge/bridge.yaml",
PollInterval: "5s", AdminAPIKeyParam: "/gobridge/prod/admin-api-key",
MonitorAPIKeyParam: "/gobridge/prod/monitor-api-key",
},
Rotation strategy: SSM values are resolved at startup and on config reload.
To rotate: (1) update the parameter value, (2) trigger reload by modifying the
EFS config file or calling POST /admin/reload. For full automation, use
Secrets Manager with a Lambda rotation function.
Set 30-day retention via LogRetention: awslogs.RetentionDays_ONE_MONTH in the
service props. The construct creates a CloudWatch log group automatically.
awslogs.NewMetricFilter(stack, jsii.String("ConfigReloadFilter"),
&awslogs.MetricFilterProps{
LogGroup: logGroup,
FilterPattern: awslogs.FilterPattern_StringValue(
jsii.String("$.msg"), jsii.String("="), jsii.String("config reload rejected"),
),
MetricNamespace: jsii.String("GoBridge/Logs"),
MetricName: jsii.String("ConfigReloadFailures"),
MetricValue: jsii.String("1"), DefaultValue: jsii.Number(0),
},
)
-- Find errors in the last hour
fields @timestamp, msg, route_id, error | filter level = "ERROR"
| sort @timestamp desc | limit 50
-- Trace a request by correlation ID
fields @timestamp, msg, route_id | filter correlation_id = "abc-123"
| sort @timestamp asc
-- Count errors by route (24h)
fields route_id | filter level = "ERROR"
| stats count(*) as error_count by route_id | sort error_count desc
Add the ADOT collector as a sidecar. It receives OTLP spans on port 4318 and forwards them to X-Ray:
adot := taskDef.AddContainer(jsii.String("adot-collector"),
&awsecs.ContainerDefinitionOptions{
Image: awsecs.ContainerImage_FromRegistry(
jsii.String("public.ecr.aws/aws-observability/aws-otel-collector:latest"), nil,
),
Command: &[]*string{jsii.String("--config=/etc/ecs/otel-config.yaml")},
},
)
adot.AddPortMappings(&awsecs.PortMapping{
ContainerPort: jsii.Number(4318), Protocol: awsecs.Protocol_TCP,
})
Observability is not configured through the bridge YAML — there is no
tracing: config key. The tracer is wired in Go code (see
Scenario 18) and honors the standard OpenTelemetry
environment variables, which is the idiomatic way to configure it per
environment in ECS:
tracer, err := oteltracing.New(ctx,
oteltracing.WithServiceName("gobridge"),
oteltracing.WithEnvironment("production"),
oteltracing.WithSamplerRatio(0.1),
// Endpoint omitted: honors OTEL_EXPORTER_OTLP_ENDPOINT from the task env.
)
Set the exporter target and resource attributes as task-definition environment variables pointing at the ADOT sidecar:
container.AddEnvironment(jsii.String("OTEL_EXPORTER_OTLP_ENDPOINT"),
jsii.String("http://localhost:4318"))
container.AddEnvironment(jsii.String("OTEL_SERVICE_NAME"),
jsii.String("gobridge"))
container.AddEnvironment(jsii.String("OTEL_RESOURCE_ATTRIBUTES"),
jsii.String("deployment.environment=production"))
Precedence is: explicit WithXxx option > OTEL_* env var > built-in default.
awsxray.NewCfnSamplingRule(stack, jsii.String("Sampling"),
&awsxray.CfnSamplingRuleProps{
SamplingRule: &awsxray.CfnSamplingRule_SamplingRuleProperty{
RuleName: jsii.String("GoBridge-Prod"), Priority: jsii.Number(100),
FixedRate: jsii.Number(0.1), ReservoirSize: jsii.Number(5),
ServiceName: jsii.String("gobridge"), ServiceType: jsii.String("*"),
Host: jsii.String("*"), HttpMethod: jsii.String("*"),
UrlPath: jsii.String("*"), ResourceArn: jsii.String("*"),
},
},
)
// Grant xray:PutTraceSegments, PutTelemetryRecords, GetSamplingRules, GetSamplingTargets
Treat the bridge config file as a versioned artifact. This scenario uses a clustered file source, so a configuration change requires cohort replacement:
flowchart LR
Repo[Git Repo] --> CP[CodePipeline]
CP --> CB[CodeBuild]
CB --> Validate[Validate Config]
Validate -->|pass| Stop[Quiesce and stop cohort]
Stop --> Write[Atomically write EFS target]
Write --> Start[Start and verify replacement cohort]
Validate -->|fail| Reject[Reject + Notify]
config/ directory.Updating an embedded document or changing ConfigFile is not a target
write. Do not delete the target as a rollout shortcut: confirmed absence requires
process exit and replacement after clustered activation, not live idle.
Uncertain teardown also exits. Read failures retain last-success processing
as degraded. Follow the
cluster rollout runbook.
The full stack listing for this scenario is on its own page: Production stack — complete CDK stack.
This stack matches the Production Single profile from the TCO Guide: approximately $80–120/month.
| Component | Monthly Estimate |
|---|---|
| Fargate (2 tasks, 0.5 vCPU / 1 GB) | ~$36 |
| VPC endpoints (6 interface) | ~$44 |
| EFS | < $1 |
| CloudWatch Logs + Metrics | ~$8 |
| SSM parameter reads | < $1 |
VPC endpoints are the second-largest cost. If your account has shared endpoints (common in enterprise landing zones), networking cost drops significantly. A single NAT Gateway (~$32/month) is an alternative when you need internet egress.