Tear a deployment down
Delete an ECS deployment completely, in an order that does not stall.
Normally you run ./teardown.sh --profile <p> --region <r>. This page is exactly the steps the script runs; when one fails, you are back here.
How it works
Section titled “How it works”| What you do | What happens |
|---|---|
Record with capture-env.sh first |
The parameters a rebuild needs are kept |
Run teardown.sh |
It prints an inventory of what would go, and the plan |
Run teardown.sh --yes |
It runs steps 1–11 below, in order |
Add --purge-retained / --purge-secrets |
What retain kept, and /af-cp/*, go too |
| Finally delete the Elastic IPs and S3 buckets yourself | Nothing of the deployment remains |
Why the order matters
Section titled “Why the order matters”Most of what a live deployment owns is not in the stacks. The Control Plane creates it at runtime:
- Workspace services, EFS access points, SSM parameters
- Under
WsRuntime=ecs-ec2, slot instances, home EBS volumes and golden snapshots
Deleting the stacks first leaves those behind and then stalls on the dependencies they hold. The order below was run end to end against two real ecs-ec2 deployments (one Persistence=delete, one =retain) and ended with every sweep counter at zero in both accounts.
1. Stop the Control Plane first
Section titled “1. Stop the Control Plane first”Everything in steps 2–7 is the Control Plane’s bookkeeping, and it launches slots on demand: a running Control Plane can recreate what you just deleted.
CL=af-af-ecs-platform # cluster = af-<platform stack name>aws ecs update-service --cluster $CL --service af-af-ecs-ingress-cp --desired-count 0teardown.sh also scales the read-aloud engine and the inference engine services to 0 here.
2. Delete the workspace services
Section titled “2. Delete the workspace services”af-ws-*, one per member that ever started a workspace. delete-service --force also removes the service’s Cloud Map entry, so do not delete that by hand (you get ServiceNotFound).
for s in $(aws ecs list-services --cluster $CL --query 'serviceArns[]' --output text | tr '\t' '\n' | grep /af-ws-); do aws ecs update-service --cluster $CL --service "$s" --desired-count 0 >/dev/null aws ecs delete-service --cluster $CL --service "$s" --force >/dev/nulldone3. Delete pool instances and home volumes
Section titled “3. Delete pool instances and home volumes”Terminating a slot takes its root volume but not the home volume. Homes are detached and kept on purpose, so they survive and keep billing.
F="Name=tag:af-pool,Values=$CL" # this deployment's pool only: its slots and engine instancesaws ec2 describe-instances --filters "$F" Name=instance-state-name,Values=pending,running,stopping,stopped \ --query 'Reservations[].Instances[].[InstanceId,State.Name,Tags[?Key==`af-role`]|[0].Value]' --output tableIDS=$(aws ec2 describe-instances --filters "$F" Name=instance-state-name,Values=pending,running,stopping,stopped \ --query 'Reservations[].Instances[].InstanceId' --output text)# check the table lists only this deployment's instances, then:[ -n "$IDS" ] && aws ec2 terminate-instances --instance-ids $IDS && aws ec2 wait instance-terminated --instance-ids $IDSaws ec2 describe-volumes --filters "$F" --query 'Volumes[].[VolumeId,State,Size]' --output table# check the home volumes listed, then:for v in $(aws ec2 describe-volumes --filters "$F" --query 'Volumes[].VolumeId' --output text); do aws ec2 delete-volume --volume-id "$v"doneThe filter is the af-pool tag, whose value is this deployment’s cluster ($CL from step 1), so nothing of another deployment in the same region is listed. It covers the slots (ecs-ec2) and the inference engine instances the Control Plane bought, as teardown.sh does. A filter on the Name tag alone (af-slot-*) would also catch every other deployment’s slots.
4. Deregister container instances that stay registered
Section titled “4. Deregister container instances that stay registered”After the EC2 instances are gone, the cluster can still list them (3 of 4 did, in one account), and the platform stack’s cluster delete fails on that. Run aws ecs deregister-container-instance --cluster $CL --container-instance <arn> --force for each one.
5. Delete the EFS access points
Section titled “5. Delete the EFS access points”List them with describe-access-points --file-system-id <efs> and delete each.
- Miss these and
delete-stack af-ecs-datastalls - The file system id is
af-ecs-data’sEfsIdoutput, notFileSystemId. A sweep script keyed on the wrong name silently deletes nothing
6. Delete SSM /af-ws/*
Section titled “6. Delete SSM /af-ws/*”The per-workspace DEK and token. /af-cp/* are the out-of-band Control Plane secrets you created before 30-ingress: keep them to redeploy into the same account, delete them to rehearse the prerequisites from scratch.
7. Delete the golden snapshots
Section titled “7. Delete the golden snapshots”List them with describe-snapshots --owner-ids self and delete each: 50 GiB per baked home, one per image tag. teardown.sh also deletes the hibernation and backup snapshots.
8. Delete the stacks in reverse order, one at a time
Section titled “8. Delete the stacks in reverse order, one at a time”Under Persistence=retain, remove the database’s deletion protection before this loop. Otherwise the data stack’s delete fails and the loop stops there. teardown.sh does the same right before it deletes the stacks.
aws rds modify-db-instance --db-instance-identifier <db> --no-deletion-protection --apply-immediatelyEmpty the template staging bucket of 20-platform (CfnTemplatesBucket) before the loop too. CloudFormation cannot delete a bucket that still holds objects, so 20-platform stops at DELETE_FAILED and the loop stops there. The bucket is not versioned, so s3 rm --recursive is enough.
CFN_BUCKET=$(aws cloudformation describe-stacks --stack-name af-ecs-platform \ --query "Stacks[0].Outputs[?OutputKey=='CfnTemplatesBucket'].OutputValue" --output text)aws s3 rm "s3://$CFN_BUCKET" --recursiveIf 20-platform still ends at DELETE_FAILED, read the stack’s events for the reason, empty the bucket again, and run the loop again: it skips the stacks already gone.
for s in af-ecs-ingress af-ecs-engines af-ecs-tts af-ecs-pool af-ecs-platform af-ecs-data af-ecs-network; do if ! err=$(aws cloudformation describe-stacks --stack-name $s 2>&1 >/dev/null); then case "$err" in *"does not exist"*) echo "skip $s (not deployed)"; continue ;; esac echo "$err" >&2; break # credentials, permission or network: stop here fi aws cloudformation delete-stack --stack-name $s aws cloudformation wait stack-delete-complete --stack-name $s || break # stop on the first failuredone- The order is
teardown.sh’s: ingress → inference engines (60-engines) → read-aloud (50-tts) → slot pool (40-ec2-pool) → platform → data → network - The engines, read-aloud and pool stacks are optional:
af-ecs-enginesexists only where inference engines were added,af-ecs-ttsonly where read-aloud was enabled,af-ecs-poolonly underecs-ec2. The loop skips only a stack CloudFormation reports as not existing (does not exist). Any otherdescribe-stackserror (expired credentials, missing permission, network) is printed and stops the loop, so nothing is reported as skipped that was not checked - The names above are the default examples. Replace them with your actual stack names (whatever you deployed
60-engines.yaml/50-tts.yaml/40-ec2-pool.yamland the rest as);teardown.shreads them from the deployment - Scale the engine service to 0 before deleting
50-tts: its Cloud Map service cannot be deleted while a task is still registered, and a task on its way down is exactly that - Wall clock: about 10 minutes (
delete) to about 16 minutes (retain, RDS snapshotting)
9. Delete what a retain deployment kept
Section titled “9. Delete what a retain deployment kept”Under Persistence=retain the data stack’s delete leaves three things behind. Delete them once af-ecs-data is gone (the deletion protection was already removed in step 8).
# after af-ecs-data is deleted:aws rds delete-db-snapshot --db-snapshot-identifier af-ecs-data-snapshot-db-<suffix> # DeletionPolicy: Snapshot leaves oneaws efs delete-file-system --file-system-id <efs> # DeletionPolicy: Retain keeps it# each EFS backup vault (DeletionPolicy: Retain) cannot be deleted until its recovery points are:aws backup list-backup-vaults --query "BackupVaultList[?starts_with(BackupVaultName,'af-ecs-data-efs-')].BackupVaultName"for rp in $(aws backup list-recovery-points-by-backup-vault --backup-vault-name <vault> \ --query 'RecoveryPoints[].RecoveryPointArn' --output text); do aws backup delete-recovery-point --backup-vault-name <vault> --recovery-point-arn "$rp"doneaws backup delete-backup-vault --backup-vault-name <vault> # once the points are goneteardown.sh removes the deletion protection before step 8, but deletes the final snapshot, the EFS and the backup vault only with --purge-retained. Without it, the vault’s recovery points expire on their own after EfsBackupRetentionDays.
10. Delete the task definitions
Section titled “10. Delete the task definitions”They cost nothing, but they outlive every stack.
list-task-definitions --family-prefix af-wsonce reported 0 while 9 ACTIVE revisions existed. Enumerate without the prefix and count- Deregister every ACTIVE revision, then
delete-task-definitionsthe INACTIVE ones in batches of 10 --max-itemswith--output textappends the pagination token (None) as a line of its own. Pass that todelete-task-definitionsand the whole batch fails. Filter withgrep '^arn:'
11. Delete the ACM validation CNAME
Section titled “11. Delete the ACM validation CNAME”30-ingress deletes the certificate and the ALB alias record, but _<hash>.<fqdn> stays in the hosted zone forever.
- Harmless, but leave it and the next deployment’s certificate validates off a record you never created, so an issuance bug goes unnoticed
- Delete it with
change-resource-record-sets. The DELETE needs the exact TTL and value; the TTL is 300
What survives a teardown
Section titled “What survives a teardown”None of the steps above deletes these.
| What survives | Why |
|---|---|
| The hosted zone | It is not the deployment’s |
The /af-cp/* secrets, if you kept them |
To redeploy into the same account. --purge-secrets deletes them |
| The NAT gateway’s Elastic IP | So a rebuild egresses from the same address (Pin the outbound IP address) |
The egress gateway’s Elastic IP (GatewayEip, when the stack allocated it for EgressRoute=gateway) |
The same (Control outbound traffic with a gateway) |
The egress record bucket (EgressArchiveBucket, with EgressArchive=create) |
So what it holds outlives the stacks |
The 60-engines models bucket |
The same. Fetching a large model from Hugging Face again takes an unpredictable time |
-
All of them are
Retainon purpose, andteardown.shtouches none of them -
An unattached EIP is billed (about $3.6 a month each) until you release it, and the buckets bill for what they hold
-
Finishing with a deployment for good is one extra deliberate step per address and bucket
Terminal window aws ec2 release-address --allocation-id eipalloc-... # each EIPaws s3 rb s3://<bucket> --force # empty then delete each bucket -
Sweep to confirm the rest is gone: EC2 instances, volumes, snapshots, AMIs and EIPs; ECS clusters and task definitions; EFS; RDS instances and manual snapshots; ALB; ACM; NAT; ECR; Cloud Map namespaces;
/aflog groups;af-*IAM roles and instance profiles; non-default VPCs
Related
Section titled “Related”- Stand up, pause and tear down with the scripts
- Back up and restore EFS
- Use idle stop and force-stop: pausing compared with deleting
For agents
Send this prompt to your agent to read this page.
Read https://agent-fleet.org/docs/operate/aws-ecs/teardown/index.md