Skip to content

Set up the alarm for an unreachable database

Get an email when the Control Plane can no longer reach its database. It looks optional, but set it as soon as the deployment is built.

What you do What happens
Nothing The Control Plane’s DB_UNAVAILABLE log line becomes the CloudWatch metric AgentFleet/<stack>/DbUnavailable. Nobody is told
Set CpAlarmEmail on 30-ingress An SNS topic and an alarm are created, and mail goes to that address
Confirm the subscription in the SNS confirmation mail When the alarm fires, the mail arrives
Redeploy 20-platform When RDS rotates its password, the Control Plane can read the new one

When the database becomes unreachable, nothing looks wrong from outside.

  • /healthz does not touch the database, so it keeps answering ok
  • So the ALB target is healthy, the ECS service is at steady state, and CloudFormation shows no drift
  • RDS rotates its managed master password on a schedule. ECS resolves a task definition’s secrets once, at task start, so a running Control Plane keeps presenting the old password

The Control Plane re-reads the secret when Postgres refuses it. This alarm tells you when even that did not work.

  1. Deploy 30-ingress with CpAlarmEmail set (Change a parameter)

    Terminal window
    --parameter-overrides CpAlarmEmail=oncall@example.com ...
  2. Confirm the subscription in the SNS confirmation mail. An unconfirmed subscription is the same as no alarm

  3. Redeploy 20-platform (below)

CpAlarmEmail defaults to empty: the metric exists and nobody is told.

30-ingress alone is not enough. The password refresh runs as CpTaskRole, which needs secretsmanager:GetSecretValue.

  • The execution role’s copy of that permission only injects the variable at start
  • Without the task-role permission the Control Plane works perfectly on the injected password. Until the next rotation

Check that the refresh has not failed with this command. No output means no problem.

Terminal window
aws logs filter-log-events --log-group-name "/af/<ingress-stack>/cp" \
--filter-pattern DB_SECRET_REFRESH_FAILED
  1. Look for DB_SECRET_REFRESH_FAILED with the command above. If it is there, CpTaskRole most likely lacks secretsmanager:GetSecretValue: redeploy 20-platform

  2. The recovery of last resort is a new deployment of the Control Plane’s service. It is blue/green, so there is no interruption, and it takes about four minutes

    Terminal window
    aws ecs update-service --cluster <cluster> --service <cp-service> --force-new-deployment

To check from outside whether the database is reachable, look at /readyz, not /healthz. /readyz asks the database and answers 503 when it cannot reach it.

For agents

Send this prompt to your agent to read this page.

Read https://agent-fleet.org/docs/operate/aws-ecs/alarms/index.md