Upgrade on ECS
Move an ECS deployment to a new release, and back to the previous one if something goes wrong. What applies to every deployment form (backups, what members see) is in Upgrade. This page is the ECS procedure.
How it works
Section titled “How it works”| What you do | What happens |
|---|---|
| Back up RDS and EFS first | You have a way back |
Run update.sh |
It checks the images, updates 20-platform and the others in order, and changes 30-ingress’s ImageTag. The Control Plane’s service is replaced blue/green |
| Nothing | Running workspaces stay on the old image, and take the new one the next time they are stopped and started |
| Run the upgrade again with the previous tag | You are back on the previous release |
The Control Plane and workspace images move together, at the same version. 30-ingress’s parameter ImageTag (default dev) is the tag of both.
Before you upgrade
Section titled “Before you upgrade”Back up
Section titled “Back up”On a production deployment, back up right before upgrading.
- Under
Persistence=retainRDS has daily automated backups, but the newest can be almost a day old - EFS has daily backups only when the data stack carries the backup vault and plan (checked next)
Take an RDS snapshot and an on-demand EFS recovery point. The EFS recovery point needs the vault, so on an older deployment do the check below first.
Check that the EFS backup exists
Section titled “Check that the EFS backup exists”For Persistence=retain deployments only. update.sh runs the same check and warns.
./efs-backup-check.sh --profile <p> --region <r> --stack af-ecs-ingress # read-onlyaws cloudformation describe-stacks --stack-name af-ecs-data \ --query "Stacks[0].Outputs[?OutputKey=='EfsBackupVaultName'].OutputValue" --output text \ --profile <p> --region <r>No output (None) for EfsBackupVaultName means the data stack is still from before v0.26.0: there is no EFS backup, whatever Persistence says.
update.sh does not touch 10-data, so turn it on by hand, with a change set and not a direct deploy.
- Create a change set with
deploy --no-execute-changeset. Parameters keep their stored values when none are given - Read it with
describe-change-set - Execute it with
execute-change-setonly if it adds the fourEfsBackup*resources and, on a data stack older than the Control Plane’s file system, the fiveCpData*resources, and nothing else. In that second case follow The Control Plane’s data root instead, which copies the readable repositories out first. AModifyorReplacementonFileSystemorDbmeans stop and do not execute: both are retained underretain, but a replacement is still a new, empty resource
A Persistence=retain deployment stood up before v0.26.0 has had no EFS backup since, and nothing else says so. After upgrading past v0.26.0, run this check once. You are done when EfsBackupVaultName appears in the data stack’s outputs and a daily recovery point exists the next morning (aws backup list-recovery-points-by-backup-vault).
Upgrade with update.sh
Section titled “Upgrade with update.sh”Use this script unless you have a reason not to. It is the hand procedure as one command, the ECS counterpart of compose’s docker compose pull && docker compose up -d.
VERSION=<v> ./update.sh --profile <p> --region <r> # deploy an already-pushed tagVERSION=<v> ./update.sh --profile <p> --region <r> --push # push to ECR firstVERSION=<v> ./update.sh --profile <p> --region <r> --dry-run| Option | Details |
|---|---|
| None | Deploy a tag already pushed to ECR |
--push |
Push to ECR first |
--dry-run |
Print the plan and change nothing |
--force |
Redeploy the Control Plane’s service even when nothing changed |
--pool-stack <stack> |
On ecs-ec2, name the slot pool stack |
What update.sh does
Section titled “What update.sh does”| What it does | Details |
|---|---|
| Warns about a missing EFS backup | Persistence=retain only. It checks with read-only calls and, when the file system is not covered, prints the change-set command that fixes it, before the deploy and again at the end. It never blocks the upgrade and never updates 10-data. Only a plain selection naming the file system’s exact ARN counts; a read it cannot make, or a selection with NotResources, Conditions or tags, is reported as unconfirmed |
| Refuses a tag that is not in ECR | CloudFormation only stores a string, so a forgotten push still deploys and the Control Plane task then dies with CannotPullContainerError. It checks both repositories first |
| Notices the empty change set | Re-pushing the same tag leaves the template unchanged, the deploy reports “No changes to deploy”, and the old image keeps running. It then falls back to ecs update-service --force-new-deployment and waits for the service to settle |
Redeploys the slot pool on ecs-ec2 |
Its user data carries security settings. It only adds a launch template version; existing slots keep the user data they were launched with. It stops when it cannot find the pool stack (the lookup needs cloudformation:ListExports); name it with --pool-stack <stack> |
| Carries on past a failed side stack, then fails | When the 50-tts or 60-engines deploy fails, it still deploys 30-ingress so the Control Plane gets the release, and ends with a FAILED: summary naming the stack and a non-zero exit status. Fix the cause from the stack events and re-run. Failures before that point (image checks, 20-platform, the slot pool) still stop the run |
| Keeps repository, image and stack in order | It deploys 20-platform (which owns the ECR repositories) first, through a change set it executes only when nothing is replaced. Then it copies af-engine-tools:<EngineToolsImageTag> from GHCR into ECR, then deploys 60-engines. In the reverse order the stack succeeds while the engines’ fetch containers sit in CannotPullContainerError. If GHCR does not have the tag either it stops: run engine-tools-image.yml first |
| Clears what went empty | A value the network or data stack no longer has (WorkspaceSubnets after a downgrade and the like) is cleared on 30-ingress and 20-platform through a change set and read back. A failed read of the network stack stops the run rather than clear anything |
| Lists the workspaces still on the old image | Nothing moves them automatically. It never stops one: stopping kills that member’s sessions, and when to take that is their call |
| Lists the running workspaces still in a private subnet | Once the network stack has workspace subnets. A warning when S3EndpointEnabled is already true. It never deploys 00-network |
| Points out a golden snapshot left behind | ecs-ec2 only. A golden baked from the previous image is not used, and new members’ homes are built empty instead (only a slower first start). With auto-bake on (the default) it is replaced within a few minutes as long as two slots are free. Otherwise re-bake with bake-golden.sh |
Upgrade by hand
Section titled “Upgrade by hand”The steps when you do not use update.sh.
-
Back up (Before you upgrade)
-
Push the new tag
Terminal window VERSION=<v> ./release-ecr.sh --profile <p> --region <r> -
Redeploy
30-ingresswith only the tag overridden. Every other parameter keeps its previous valueTerminal window CFN_BUCKET=$(aws cloudformation describe-stacks --stack-name <platform stack> --query "Stacks[0].Outputs[?OutputKey=='CfnTemplatesBucket'].OutputValue" --output text --profile <p> --region <r>)aws cloudformation deploy --stack-name af-ecs-ingress \--template-file cfn/30-ingress.yaml --s3-bucket "$CFN_BUCKET" --s3-prefix cfn --capabilities CAPABILITY_NAMED_IAM \--parameter-overrides ImageTag=<v> \--profile <p> --region <r>
What happens after the upgrade
Section titled “What happens after the upgrade”- The Control Plane and Console service moves to the new image: a brief blue/green replacement behind the ALB
- Workspaces are not touched. Each one takes the new image on its next start; a running workspace keeps its task until it is stopped and started
- Nothing changes on EFS or RDS, and state carries over
Members see “A new version is available” in the Console and “Restart needed” on the workspace bar. The difference is in Upgrade.
When “Restart needed” never appears
Section titled “When “Restart needed” never appears”The Control Plane shows “Restart needed” by comparing, through ECR, the image a workspace started with against what the tag points at now. It never compares version numbers.
- That needs
ecr:BatchGetImage/ecr:DescribeImageson the workspace repository inCpTaskRole. On an older deployment, redeploy20-platformonce. Without the permission the comparison fails silently and the badge never appears - A deployment whose
AF_ECS_WORKSPACE_IMAGEis not an ECR image (pulling from GHCR directly, say) gets no badge
Roll back
Section titled “Roll back”The same operation with the previous tag rolls back.
- Never push a different image under a release tag that is already out
- The repositories are
MUTABLE, so nothing stops an overwrite. That is what lets:devbe overwritten for development; leaving release tags alone is a rule you keep
00-network and 10-data are not upgraded
Section titled “00-network and 10-data are not upgraded”00-network and 10-data are part of neither this procedure nor update.sh. A release that needs either says so in its release notes. The Control Plane’s data root, Keep workspace tasks off NFS and Upgrade from v0.34.x below are three.
Clear a parameter
Section titled “Clear a parameter”aws cloudformation deploy --parameter-overrides Key= did not clear Key on a live stack (“No changes”). The CLI does send the empty value; the cause is not known.
update.sh and standup.sh clear what they need to themselves and read it back. To clear one by hand, use clear-params.sh.
./clear-params.sh --profile <p> --region <r> --stack <stack> <Key>...It builds a change set on the stack’s current template that leaves the keys out. A key becomes empty only when its Default in the template is exactly "", and every other parameter keeps its previous value.
| How you run it | Details |
|---|---|
| No option | Builds the change set, prints the resource changes and which keys go empty, keeps the change set, prints the command that executes it, and stops |
--execute <change-set-name> <Key>... |
Run it when what was printed is what you want. Refuses a change set that is gone, not AVAILABLE, or not exactly the clear of those keys (and deletes such a change set unless it is gone or already running). Reads each value back afterwards |
--yes |
Builds, prints and executes in one go |
--dry-run |
Builds, prints and deletes the change set |
- A masked (
NoEcho) parameter passes only asUsePreviousValue, and an SSM-typed one only if it resolves to the same value. Anything else it cannot prove unchanged is refused - A change set that fails to build (“didn’t contain changes”, for example) is reported and deleted
ecs-ec2: slot AMIs follow Amazon’s SSM parameters
Section titled “ecs-ec2: slot AMIs follow Amazon’s SSM parameters”With WsRuntime=ecs-ec2 the Control Plane reads Amazon’s public ECS-optimized AL2023 AMI parameters (x86_64 and arm64) itself, about hourly, and launches every slot with the result. Slots follow Amazon’s updates without a deploy.
SlotAmiId,SlotAmiIdArm64andEc2SlotAmiArm64are the initial and fallback values- The Control Plane’s role needs
ssm:GetParameteronarn:aws:ssm:<region>::parameter/aws/service/ecs/optimized-ami/amazon-linux-2023/*andec2:DescribeLaunchTemplateVersions. Both are in20-platform: deploy it before or with the new Control Plane - Until then the Control Plane falls back quietly (one log line) and never marks a slot outdated because of its AMI
- The first new AMI after the upgrade makes every slot outdated at once; they are replaced at their next start
How outdated slots are replaced is in Replace and stop slots.
The Control Plane’s data root
Section titled “The Control Plane’s data root”The Control Plane keeps the internal git provider’s repositories and their LFS objects under its data root (WS_DATA). Up to v0.45.x that was /tmp on the Control Plane task’s own disk, so every replacement of the task (an upgrade, a restart, a failed task) emptied it. Later releases mount a file system of its own instead.
| What | Where |
|---|---|
| File system | 10-data’s CpDataFileSystem: encrypted, elastic throughput, Retain / Delete following Persistence, in the daily backup under retain (Back up and restore EFS) |
| Who can mount it | The Control Plane task only, through its one access point (/cp-data), with TLS and IAM authorization. Its mount targets take NFS only from the Control Plane’s security group, and its policy admits no anonymous client |
| Inside the task | /var/lib/af-cp, which is WS_DATA. 30-ingress also sets AF_DATA_DURABLE_REQUIRED=1 |
Upgrade from v0.45.x or earlier
Section titled “Upgrade from v0.45.x or earlier”Until the upgrade, the repositories are on the running Control Plane task’s own disk, and the upgrade replaces that task. Two kinds of repository are listed in the Console now:
| Kind | How to tell | What happens |
|---|---|---|
| Created or pushed to since the task last started | git ls-remote <clone URL> answers |
Still readable. Its only copy goes with the old task unless you copy it out first (step 3) |
| Older than the task | git ls-remote fails (“not found”) |
Already empty; only its row remains |
10-data has to be updated before update.sh: 30-ingress imports the new file system, and update.sh stops with “no readable CpDataEfsId output” until it can read it. Nothing is changed when it stops.
-
Back up (Before you upgrade)
-
Announce a window. Ask members not to create, rename or push to internal repositories until step 7. Nothing enforces it: a push after its repository was copied in step 3 is lost, and during the deploy the old and new task both serve
/git/for a moment -
Copy out every repository that is still readable. A tenant administrator of each tenant does it in one of their own workspaces in that tenant: git there is already set up to authenticate to the internal git, which an outside machine is not (it would only see authentication errors). Work in a directory under the home, which survives a workspace restart, and use the clone URLs the Console lists (Settings > “Internal repos”). Every failure stops the function with a non-zero status; nothing is classified as lost unless the Control Plane answered “not found”. Before running one again, remove its
<name>.gitTerminal window refs() { awk '$2 != "HEAD" && $2 !~ /\^\{\}$/' | sort; } # what ls-remote lists, minus HEAD and peeled tagsrescue() { # rescue <name> <clone URL>local name="$1" url="$2" outif ! out="$(git ls-remote "$url" 2>&1)"; thencase "$out" in*"not found"*) echo "$name: lost"; return 0 ;;*) echo "$name: UNCONFIRMED: $out" >&2; return 1 ;;esacfiprintf '%s\n' "$out" | refs > "$name.refs"git clone --mirror "$url" "$name.git" || return 1git -C "$name.git" lfs fetch --all || return 1git -C "$name.git" for-each-ref --format='%(objectname)%09%(refname)' | sort | diff "$name.refs" - || return 1echo "$name: copied"}rescue <name> <clone URL> # once per listed repositoryCheckpoint: go on to step 4 only when every listed repository printed
copiedorlost. AnUNCONFIRMEDone (an authentication error, a timeout, a 5xx) or one that stopped on an error is neither: find out why and run it again. Until then nothing has been changed, and the old task still holds the only copy -
Update
10-datathrough a change set, with the parameters it already has. The same--profileand--regiongo on every commandTerminal window aws cloudformation deploy --stack-name af-ecs-data --template-file cfn/10-data.yaml \--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM --no-execute-changeset \--profile <p> --region <r>aws cloudformation describe-change-set --change-set-name <the ARN it printed> \--query 'Changes[].ResourceChange.[Action,LogicalResourceId,Replacement]' --output text \--profile <p> --region <r>- Expect Add of
CpDataEfsSg,CpDataFileSystem,CpDataAccessPoint,CpDataMountTarget0andCpDataMountTarget1 - Under
retainwith backups already on, also ModifyEfsBackupSelection, possibly as a replacement. That is harmless: the recovery points stay in the vault - Under
retainon a data stack from before v0.26.0 (noEfsBackupVaultNameoutput), also Add of the fourEfsBackup*resources (Turn backups on for a running deployment) - Anything on
FileSystemorDb, or anything else: stop and do not execute - Otherwise execute it and wait
Terminal window aws cloudformation execute-change-set --change-set-name <the ARN> --profile <p> --region <r>aws cloudformation wait stack-update-complete --stack-name af-ecs-data --profile <p> --region <r> - Expect Add of
-
Run
update.shas usual. The new Control Plane task starts on the empty file system -
Clear the old rows. In the Console, delete every repository that printed
copiedorlostin step 3: the new file system has none of them, and creating a repository under a name the database still lists is refused (“already exists”). Deleting one also releases the LFS bytes it counted toward the tenant’s limit -
Put back what you copied. In the same workspace, create each
copiedrepository again under its old name, then push the copy. The function stops unless the refs the Control Plane lists afterwards, names and object IDs, equal the ones saved in step 3. A repository created in the Console starts onmain; if the old default branch was another one, tell its membersTerminal window refs() { awk '$2 != "HEAD" && $2 !~ /\^\{\}$/' | sort; }putback() { # putback <name> <clone URL>local name="$1" url="$2"git -C "$name.git" lfs push --all "$url" || return 1git -C "$name.git" push --mirror "$url" || return 1git ls-remote "$url" | refs | diff "$name.refs" - || return 1echo "$name: restored"}putback <name> <clone URL>
A repository that printed lost cannot be recovered from the deployment. A member’s own clone, if one exists, can be pushed into a re-created repository the same way.
Go back
Section titled “Go back”Taking 30-ingress back to v0.45.x puts WS_DATA back on /tmp. What was pushed in the meantime stays on CpDataFileSystem but is not served, and anything pushed after going back is on the task’s disk again. The 10-data additions can stay.
ecs-ec2: keep workspace tasks off NFS
Section titled “ecs-ec2: keep workspace tasks off NFS”A member’s sign-in data is on EFS, under an access point of their own. The file system cannot tell one member’s task from another’s, so a workspace that learned another member’s access point ID could mount it. From this release the slot mounts the access points and the task only receives them as directories, which makes it possible to close the file system to workspace tasks altogether. On Fargate (WsRuntime=ecs) this cannot be done: leave EfsWorkspaceIngress at allow there.
It takes two steps, and the second one is yours to time.
-
Update
10-databeforeupdate.sh, through a change set, with the parameters it already has. It adds a security group for the home task, a rule on the file system’s security group that admits it, and two outputs; nothing is replaced or removed.update.shstops with “no readable HomeOpsSgId / EfsSgId output” until it can read them, and nothing is changed when it stops. Then runupdate.shas usual; it adds the slots’ NFS rule with40-ec2-pool. -
Set
EfsWorkspaceIngress=denyon10-dataonce every workspace has been stopped and started on the new release. A workspace still running from before the upgrade mounts EFS through its own network interface and loses its sign-in data when the rule goes, so check first.Terminal window aws cloudformation deploy --stack-name af-ecs-data --template-file cfn/10-data.yaml \--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \--parameter-overrides EfsWorkspaceIngress=deny --no-execute-changesetReview the change set (it modifies the file system’s security group and nothing else), then execute it.
| If | Then |
|---|---|
| A Start fails with “mounting access point … failed” in the Control Plane’s log | The slots cannot reach the mount targets. 40-ec2-pool was not updated, or it names another data stack (DataStackName). Nothing is quarantined; start again after fixing it |
| You go back to a release from before this one | Set EfsWorkspaceIngress=allow first. The old release mounts EFS from the task |
| A member’s workspace was never restarted and step 2 is done | It keeps running until its NFS connection is reopened, then loses ~/.claude and its logins until it is restarted. Restart it |
The details are in PARAMETERS.md, “EFS: who may reach the mount targets”.
Upgrade from v0.34.x
Section titled “Upgrade from v0.34.x”v0.34.x’s 00-network has an S3 gateway endpoint without a policy and no workspace subnets. The current release gives workspaces subnets of their own and the endpoint a policy that refuses every S3 request it does not list (AccessDenied, with no NAT fallback), a workspace’s own S3 included while that workspace still runs in a private subnet.
update.sh does not deploy 00-network, so this is done by hand, in a maintenance window in which every running workspace is restarted.
Do not deploy the new 00-network with its defaults. S3EndpointEnabled defaults to true: the change set is Add of the six workspace resources plus Modify S3Endpoint (its policy), and executing it refuses the S3 requests of every workspace still in a private subnet at once.
-
Back up (Before you upgrade)
-
Deploy
00-networkwithS3EndpointEnabled=false, through a change set. The stack now needsCAPABILITY_IAM(it declares the egress gateway’s instance role, built only withEgressRoute=gateway)Terminal window aws cloudformation deploy --stack-name af-ecs-network --template-file cfn/00-network.yaml \--capabilities CAPABILITY_IAM --parameter-overrides S3EndpointEnabled=false \--no-execute-changeset --profile <p> --region <r>aws cloudformation describe-change-set --change-set-name <the ARN it printed> \--query 'Changes[].ResourceChange.[Action,LogicalResourceId,Replacement]' --output text- Expect Add of six resources,
WorkspaceSubnet0,WorkspaceSubnet1,WorkspaceRt,WorkspaceDefaultRoute,WorkspaceAssoc0andWorkspaceAssoc1, and RemoveS3Endpoint - The removal is expected. Until step 5 the private subnets’ S3 (image pulls on
ecs-ec2, model fetches) goes through the NAT gateway and pays its data processing - Anything else added, modified or replaced: stop
- Otherwise run
execute-change-set --change-set-name <ARN>and wait withwait stack-update-complete --stack-name af-ecs-network
- Expect Add of six resources,
-
Run
update.sh(VERSION=<v> ./update.sh --profile <p> --region <r>). It passes the network stack’sWorkspaceSubnetsto30-ingress, so every start from then on lands in a workspace subnet. It ends by listing the running workspaces still in a private subnet:WARNING:when there are any, an explicit “none” when the check ran and found none -
Stop and start every running workspace (that list, or
pause.sh --status). A running task keeps its ENI. Check each one with the command below, which names a workspace subnet.update.sh --dry-runrepeats the list and changes nothingTerminal window aws ecs describe-tasks --cluster <c> --tasks <t> --query "tasks[].attachments[].details[?name=='subnetId'].value" -
Deploy
00-networkwithS3EndpointEnabled=true, again through a change set. The change is one Add,S3Endpoint. Private-subnet S3 moves from the NAT route to the endpoint and open TCP connections do not survive that, so do it when no large model fetch or image pull is running
Go back to v0.34.x
Section titled “Go back to v0.34.x”| How far you got | How to go back |
|---|---|
| Before step 5 | The private subnets’ S3 already goes through the NAT |
| After step 5 | S3EndpointEnabled=false is Remove S3Endpoint only |
Taking 00-network back to v0.34.x’s template |
It removes the six resources and modifies S3Endpoint back to no policy. First stop every workspace that runs in a workspace subnet; then update.sh clears 30-ingress’s WorkspaceSubnets (the ids are gone) |
Going back to v0.34.x also drops the parameters v0.34.x does not know, GatewayEipAllocationId and GatewayEipAddress among them. The next switch to EgressRoute=gateway must pass both again.
Related
Section titled “Related”- Upgrade: what applies to every deployment form
- Build on ECS
- Keep templates under the size limit
- Replace and stop slots
- Troubleshoot ECS
For agents
Send this prompt to your agent to read this page.
Read https://agent-fleet.org/docs/operate/aws-ecs/upgrade/index.md