Skip to content

Troubleshoot the slot pool

On an ecs-ec2 deployment, find the cause and the fix from the symptom when slots or homes misbehave. Build and start problems common to every ECS deployment are in Troubleshoot ECS.

What you do What you learn or what happens
Look at Admin → Slots Each slot’s state, quarantined slots and slots that do not answer, and the golden snapshot’s status
Remove a quarantined slot with Terminate The instance and its root volume are deleted, and the billing stops
Stop a slot that does not answer with Stop (force) The instance stops. The next start brings the same slot back healthy, or, if the container instance had to be deregistered, moves the home to a new slot
Find the symptom in the tables below The cause, and which page’s steps fix it

How to read the page is in Replace and stop slots.

A slot that cannot mount a home (quarantine)

Section titled “A slot that cannot mount a home (quarantine)”

A slot that could not mount a home is quarantined by the Control Plane. This is for failures no retry fixes, such as a wedged kernel holding a deleted volume’s NVMe namespace.

What quarantine does What it means
Takes it out of the pool Its tag becomes af-role=quarantined, and nobody else lands on it
Detaches its home The home is free to use another slot
Stops the instance The root volume stays and keeps billing

Nothing removes a quarantined slot by itself. It stays on the Slots page with its reason.

  1. Open Admin → Slots. With quarantined slots, a count appears at the top and their rows say quarantined
  2. If there is anything to examine, do it first. The instance and its root volume cannot be brought back once terminated
  3. Press Terminate at the end of the row, check the reason in the confirmation and terminate it
  • The quarantine reason goes into the audit log before the instance and its tags disappear
  • This button is the only route in the product that deletes a machine on request (DELETE /api/admin/ec2-pool/slots/{id}, super_admin)
  • It accepts only an instance of this pool tagged af-role=quarantined, with no home attached and no live claim. A working slot is refused; it leaves through Ec2SlotTerminateAfterSec instead

EC2 says it is running and its status checks are ok, but the ECS agent is not connected, SSM is silent, and a home is still attached. Stop it with Stop (force). The steps and the refusal codes are in Stop a slot that does not answer.

How to notice one before anybody starts is in Notice a slot that does not answer.

Symptom Cause and fix
The Control Plane does not boot after switching to ecs-ec2 Ec2SlotLaunchTemplate is empty. Pass the SlotLaunchTemplateId output of 40-ec2-pool (Build the pool)
The log says slot class "…" is arm64 but AF_ECS_EC2_AMI_ARM64 is empty and it does not boot An arm64 class is declared and Ec2SlotAmiArm64 is empty (Add a Graviton (arm64) class)
A class you wrote does not appear Its arch is neither x86_64 nor arm64, or it has no usable rung. Such a class is dropped (Write the classes)
Symptom Cause and fix
The start fails with slot pool is full The pool is at Ec2MaxSlots and no dormant slot can be reclaimed: everything is running, or claimed by a start in flight. Raise Ec2MaxSlots (Three limits)
The start fails with no configured AZ (…) could take a new … slot No AZ could launch an instance of that type (capacity or quota). Try later, or use another class
Only one user can never start No slot can run in the AZ their home is in. A home cannot leave its AZ (Move or lose an AZ)
The start fails with no workspace subnet in … (AF_ECS_WORKSPACE_SUBNETS) An AZ of AF_ECS_SUBNETS has no workspace subnet. Add one for every AZ
Only tasks landing in one AZ never come up That AZ has no EFS mount target, so the logins cannot be mounted (Before you use several AZs)
On an arm64 class the slot launches but the task does not run The workspace image has no arm64 manifest (Add a Graviton (arm64) class)
A start sits at starting for a long time, then the workspace is rebuilt on another slot The slot was EC2-running but the cluster could not reach its agent. After AF_ECS_EC2_SLOT_LOST_AFTER_SEC (default 300 seconds) the Control Plane gives up on it and rebuilds elsewhere
A member moved to a class of another architecture takes minutes longer to start Architecture-dependent tools are being reinstalled. The member reinstalls what they built under ~/repos (Move a member to a class of another architecture)
Symptom Cause and fix
A new member’s first start is slow No golden snapshot is used, and the home is built empty. The reason is in the Golden snapshot part of Slots (Golden snapshot)
The golden part keeps waiting for free slots A bake needs two free slots. If you baked by hand and did not destroy the seed, destroy it (Bake it by hand)
The golden part says it is not used because it was baked from another image The image changed. On a deployment that bakes automatically, wait. With AF_ECS_EC2_GOLDEN_AUTOBAKE=0, re-bake with bake-golden.sh
The slot I/O alarms fire A workspace used up its instance’s memory and the instance is thrashing. If Ec2HostReserveMb is off, put it back to auto, and force-stop a slot that does not answer (What a workspace actually gets)
A start from a hibernated home is slow It is being restored from the snapshot. That start takes about 2 minutes longer, and the disk is slower for a few hours afterwards
Symptom Cause and fix
The EBS bill does not go down when nobody is working Ec2SlotTerminateAfterSec is 0, so stopped slots’ root volumes are kept up to Ec2MaxSlots. Set it to 14400 (Three limits)
Homes nobody uses keep costing Hibernation is off. Set it per tenant or as the deployment default (Hibernate unused homes)
Symptom Cause and fix
Neither hibernation nor spare copies ever happen AF_IDLE_SWEEP_INTERVAL=0. Both are triggered by the idle-stop sweeper
One tenant’s homes are never hibernated That tenant has 0 (Set them per tenant)
The Spare copy column stays at none Neither the tenant nor the deployment default (AF_ECS_EC2_BACKUP_EVERY_SEC) sets an interval (Keep a spare copy of each home)
A removed member’s copies stay Clean home does not delete them. Delete backups or destroying the workspace does (Remove a member)
After losing an AZ, users with a home there cannot start Take the failed AZ out first, then rebuild from a spare copy if it does not come back (When you lose an AZ, Restore a home from a spare copy)

For agents

Send this prompt to your agent to read this page.

Read https://agent-fleet.org/docs/operate/aws-ecs/slot-troubleshooting/index.md