Skip to content

Tear a deployment down

Delete an ECS deployment completely, in an order that does not stall. Normally you run ./teardown.sh --profile <p> --region <r>. This page is exactly the steps the script runs; when one fails, you are back here.

What you do What happens
Record with capture-env.sh first The parameters a rebuild needs are kept
Run teardown.sh It prints an inventory of what would go, and the plan
Run teardown.sh --yes It runs steps 1–11 below, in order
Add --purge-retained / --purge-secrets What retain kept, and /af-cp/*, go too
Finally delete the Elastic IPs and S3 buckets yourself Nothing of the deployment remains

Most of what a live deployment owns is not in the stacks. The Control Plane creates it at runtime:

  • Workspace services, EFS access points, SSM parameters
  • Under WsRuntime=ecs-ec2, slot instances, home EBS volumes and golden snapshots

Deleting the stacks first leaves those behind and then stalls on the dependencies they hold. The order below was run end to end against two real ecs-ec2 deployments (one Persistence=delete, one =retain) and ended with every sweep counter at zero in both accounts.

Everything in steps 2–7 is the Control Plane’s bookkeeping, and it launches slots on demand: a running Control Plane can recreate what you just deleted.

Terminal window
CL=af-af-ecs-platform # cluster = af-<platform stack name>
aws ecs update-service --cluster $CL --service af-af-ecs-ingress-cp --desired-count 0

teardown.sh also scales the read-aloud engine and the inference engine services to 0 here.

af-ws-*, one per member that ever started a workspace. delete-service --force also removes the service’s Cloud Map entry, so do not delete that by hand (you get ServiceNotFound).

Terminal window
for s in $(aws ecs list-services --cluster $CL --query 'serviceArns[]' --output text | tr '\t' '\n' | grep /af-ws-); do
aws ecs update-service --cluster $CL --service "$s" --desired-count 0 >/dev/null
aws ecs delete-service --cluster $CL --service "$s" --force >/dev/null
done

Terminating a slot takes its root volume but not the home volume. Homes are detached and kept on purpose, so they survive and keep billing.

Terminal window
F="Name=tag:af-pool,Values=$CL" # this deployment's pool only: its slots and engine instances
aws ec2 describe-instances --filters "$F" Name=instance-state-name,Values=pending,running,stopping,stopped \
--query 'Reservations[].Instances[].[InstanceId,State.Name,Tags[?Key==`af-role`]|[0].Value]' --output table
IDS=$(aws ec2 describe-instances --filters "$F" Name=instance-state-name,Values=pending,running,stopping,stopped \
--query 'Reservations[].Instances[].InstanceId' --output text)
# check the table lists only this deployment's instances, then:
[ -n "$IDS" ] && aws ec2 terminate-instances --instance-ids $IDS && aws ec2 wait instance-terminated --instance-ids $IDS
aws ec2 describe-volumes --filters "$F" --query 'Volumes[].[VolumeId,State,Size]' --output table
# check the home volumes listed, then:
for v in $(aws ec2 describe-volumes --filters "$F" --query 'Volumes[].VolumeId' --output text); do
aws ec2 delete-volume --volume-id "$v"
done

The filter is the af-pool tag, whose value is this deployment’s cluster ($CL from step 1), so nothing of another deployment in the same region is listed. It covers the slots (ecs-ec2) and the inference engine instances the Control Plane bought, as teardown.sh does. A filter on the Name tag alone (af-slot-*) would also catch every other deployment’s slots.

4. Deregister container instances that stay registered

Section titled “4. Deregister container instances that stay registered”

After the EC2 instances are gone, the cluster can still list them (3 of 4 did, in one account), and the platform stack’s cluster delete fails on that. Run aws ecs deregister-container-instance --cluster $CL --container-instance <arn> --force for each one.

List them with describe-access-points --file-system-id <efs> and delete each.

  • Miss these and delete-stack af-ecs-data stalls
  • The file system id is af-ecs-data’s EfsId output, not FileSystemId. A sweep script keyed on the wrong name silently deletes nothing

The per-workspace DEK and token. /af-cp/* are the out-of-band Control Plane secrets you created before 30-ingress: keep them to redeploy into the same account, delete them to rehearse the prerequisites from scratch.

List them with describe-snapshots --owner-ids self and delete each: 50 GiB per baked home, one per image tag. teardown.sh also deletes the hibernation and backup snapshots.

8. Delete the stacks in reverse order, one at a time

Section titled “8. Delete the stacks in reverse order, one at a time”

Under Persistence=retain, remove the database’s deletion protection before this loop. Otherwise the data stack’s delete fails and the loop stops there. teardown.sh does the same right before it deletes the stacks.

Terminal window
aws rds modify-db-instance --db-instance-identifier <db> --no-deletion-protection --apply-immediately

Empty the template staging bucket of 20-platform (CfnTemplatesBucket) before the loop too. CloudFormation cannot delete a bucket that still holds objects, so 20-platform stops at DELETE_FAILED and the loop stops there. The bucket is not versioned, so s3 rm --recursive is enough.

Terminal window
CFN_BUCKET=$(aws cloudformation describe-stacks --stack-name af-ecs-platform \
--query "Stacks[0].Outputs[?OutputKey=='CfnTemplatesBucket'].OutputValue" --output text)
aws s3 rm "s3://$CFN_BUCKET" --recursive

If 20-platform still ends at DELETE_FAILED, read the stack’s events for the reason, empty the bucket again, and run the loop again: it skips the stacks already gone.

Terminal window
for s in af-ecs-ingress af-ecs-engines af-ecs-tts af-ecs-pool af-ecs-platform af-ecs-data af-ecs-network; do
if ! err=$(aws cloudformation describe-stacks --stack-name $s 2>&1 >/dev/null); then
case "$err" in *"does not exist"*) echo "skip $s (not deployed)"; continue ;; esac
echo "$err" >&2; break # credentials, permission or network: stop here
fi
aws cloudformation delete-stack --stack-name $s
aws cloudformation wait stack-delete-complete --stack-name $s || break # stop on the first failure
done
  • The order is teardown.sh’s: ingress → inference engines (60-engines) → read-aloud (50-tts) → slot pool (40-ec2-pool) → platform → data → network
  • The engines, read-aloud and pool stacks are optional: af-ecs-engines exists only where inference engines were added, af-ecs-tts only where read-aloud was enabled, af-ecs-pool only under ecs-ec2. The loop skips only a stack CloudFormation reports as not existing (does not exist). Any other describe-stacks error (expired credentials, missing permission, network) is printed and stops the loop, so nothing is reported as skipped that was not checked
  • The names above are the default examples. Replace them with your actual stack names (whatever you deployed 60-engines.yaml / 50-tts.yaml / 40-ec2-pool.yaml and the rest as); teardown.sh reads them from the deployment
  • Scale the engine service to 0 before deleting 50-tts: its Cloud Map service cannot be deleted while a task is still registered, and a task on its way down is exactly that
  • Wall clock: about 10 minutes (delete) to about 16 minutes (retain, RDS snapshotting)

Under Persistence=retain the data stack’s delete leaves three things behind. Delete them once af-ecs-data is gone (the deletion protection was already removed in step 8).

Terminal window
# after af-ecs-data is deleted:
aws rds delete-db-snapshot --db-snapshot-identifier af-ecs-data-snapshot-db-<suffix> # DeletionPolicy: Snapshot leaves one
aws efs delete-file-system --file-system-id <efs> # DeletionPolicy: Retain keeps it
# each EFS backup vault (DeletionPolicy: Retain) cannot be deleted until its recovery points are:
aws backup list-backup-vaults --query "BackupVaultList[?starts_with(BackupVaultName,'af-ecs-data-efs-')].BackupVaultName"
for rp in $(aws backup list-recovery-points-by-backup-vault --backup-vault-name <vault> \
--query 'RecoveryPoints[].RecoveryPointArn' --output text); do
aws backup delete-recovery-point --backup-vault-name <vault> --recovery-point-arn "$rp"
done
aws backup delete-backup-vault --backup-vault-name <vault> # once the points are gone

teardown.sh removes the deletion protection before step 8, but deletes the final snapshot, the EFS and the backup vault only with --purge-retained. Without it, the vault’s recovery points expire on their own after EfsBackupRetentionDays.

They cost nothing, but they outlive every stack.

  • list-task-definitions --family-prefix af-ws once reported 0 while 9 ACTIVE revisions existed. Enumerate without the prefix and count
  • Deregister every ACTIVE revision, then delete-task-definitions the INACTIVE ones in batches of 10
  • --max-items with --output text appends the pagination token (None) as a line of its own. Pass that to delete-task-definitions and the whole batch fails. Filter with grep '^arn:'

30-ingress deletes the certificate and the ALB alias record, but _<hash>.<fqdn> stays in the hosted zone forever.

  • Harmless, but leave it and the next deployment’s certificate validates off a record you never created, so an issuance bug goes unnoticed
  • Delete it with change-resource-record-sets. The DELETE needs the exact TTL and value; the TTL is 300

None of the steps above deletes these.

What survives Why
The hosted zone It is not the deployment’s
The /af-cp/* secrets, if you kept them To redeploy into the same account. --purge-secrets deletes them
The NAT gateway’s Elastic IP So a rebuild egresses from the same address (Pin the outbound IP address)
The egress gateway’s Elastic IP (GatewayEip, when the stack allocated it for EgressRoute=gateway) The same (Control outbound traffic with a gateway)
The egress record bucket (EgressArchiveBucket, with EgressArchive=create) So what it holds outlives the stacks
The 60-engines models bucket The same. Fetching a large model from Hugging Face again takes an unpredictable time
  • All of them are Retain on purpose, and teardown.sh touches none of them

  • An unattached EIP is billed (about $3.6 a month each) until you release it, and the buckets bill for what they hold

  • Finishing with a deployment for good is one extra deliberate step per address and bucket

    Terminal window
    aws ec2 release-address --allocation-id eipalloc-... # each EIP
    aws s3 rb s3://<bucket> --force # empty then delete each bucket
  • Sweep to confirm the rest is gone: EC2 instances, volumes, snapshots, AMIs and EIPs; ECS clusters and task definitions; EFS; RDS instances and manual snapshots; ALB; ACM; NAT; ECR; Cloud Map namespaces; /af log groups; af-* IAM roles and instance profiles; non-default VPCs

For agents

Send this prompt to your agent to read this page.

Read https://agent-fleet.org/docs/operate/aws-ecs/teardown/index.md