Build on ECS
Build Agent Fleet on AWS ECS, start the Control Plane, and get to the point where you can sign in with your IdP. The stacks and their order are in Understand the ECS layout.
How it works
Section titled “How it works”| What you do | What happens |
|---|---|
| Once per account, create the service-linked roles. If you will split spend, also turn on billing access in Billing | ECS clusters can be created. The Control Plane can read spend |
Deploy 00-network, 10-data and 20-platform |
The network, EFS and RDS, ECR, the cluster and the IAM roles are built |
| Push the images to ECR | ECS can pull them |
If you use them, deploy 40-ec2-pool, 50-tts and 60-engines |
The slot pool, speech and your own inference engines are built |
| Put the Control Plane’s secrets in SSM and register the redirect URI with your IdP | The Control Plane can read its secrets, and the IdP can send sign-ins back |
Deploy 30-ingress |
The certificate, ALB, DNS and Control Plane service are built, and you can sign in to the Console |
| If you split spend, start one workspace on the day you build | The Control Plane writes the tags; once AWS discovers the keys (up to 24 hours), the Control Plane activates the cost allocation tags |
Run the commands from deploy/aws/ecs/ in the repository. standup.sh builds in the same order, but it rebuilds from an existing deployment’s parameters that capture-env.sh saved (params/30-ingress and the rest). In a new environment without them it exits at once, so build a first deployment with the steps on this page.
Before you start (once per account)
Section titled “Before you start (once per account)”Create the service-linked roles
Section titled “Create the service-linked roles”A fresh account has no AWSServiceRoleForECS, and creating a cluster with a Service Connect default namespace fails with “ECS Service Linked Role is not ready”.
If you run self-hosted inference engines (60-engines), also create AWSServiceRoleForEC2Fleet, which AWS asks for before CreateFleet. Calls have gone through in an account without it, but it is cheap insurance.
standup.sh creates both. By hand, as below. It is idempotent: ignore the “has been taken” error on a re-run.
aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com || trueaws iam create-service-linked-role --aws-service-name ec2fleet.amazonaws.com || true# and, for a deployment that declares a `spot` offer:aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || trueFor a deployment that declares a spot offer, run the third line too.
Turn on billing access (optional)
Section titled “Turn on billing access (optional)”If you will split spend per member, turn on “IAM user and role access to Billing information” in the Billing console of the payer account. The templates cannot set it.
While it is off, every ce:GetCostAndUsage call by CpTaskRole fails even though the policy is attached.
Activating the cost allocation tags comes after the build (Split spend per member).
Deploy the base stacks
Section titled “Deploy the base stacks”Capabilities each stack needs
Section titled “Capabilities each stack needs”| Stack | --capabilities |
Why |
|---|---|---|
00-network |
CAPABILITY_IAM |
It declares the egress gateway’s instance role |
10-data |
CAPABILITY_AUTO_EXPAND CAPABILITY_IAM |
It declares Transform: AWS::LanguageExtensions. Under Persistence=retain it also creates the unnamed IAM role AWS Backup uses for the EFS backups |
20-platform |
CAPABILITY_NAMED_IAM |
It creates named IAM roles |
40-ec2-pool, 60-engines, 30-ingress |
CAPABILITY_NAMED_IAM |
Each attaches a named managed policy to the Control Plane’s task role |
Without them the change set is refused with InsufficientCapabilitiesException … Requires capabilities : [CAPABILITY_IAM] or similar.
Deploy in order
Section titled “Deploy in order”-
Deploy
00-networkTerminal window aws cloudformation deploy --stack-name af-ecs-network --template-file cfn/00-network.yaml \--capabilities CAPABILITY_IAM \--profile af-sandbox --region ap-northeast-1 -
Deploy
10-data. For a deployment whose data must survive, usePersistence=retainTerminal window aws cloudformation deploy --stack-name af-ecs-data \--template-file cfn/10-data.yaml --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM \--parameter-overrides Persistence=retain \--profile af-sandbox --region ap-northeast-1 -
Deploy
20-platformTerminal window aws cloudformation deploy --stack-name af-ecs-platform \--template-file cfn/20-platform.yaml --capabilities CAPABILITY_NAMED_IAM \--profile af-sandbox --region ap-northeast-1
Persistence defaults to delete: deleting the stack deletes EFS and RDS with it.
Push the images to ECR
Section titled “Push the images to ECR”The ECR repositories belong to 20-platform, so deploy it first. Creating the repositories out of band makes the later 20-platform deploy fail with AlreadyExists.
release-ecr.sh only checks that the repositories exist; it does not create them.
# images already local (built by deploy/release/build.sh):VERSION=<v> ./release-ecr.sh --profile af-sandbox --region <region># or from the air-gap images tar (B) when the build host is a different machine:VERSION=<v> ./release-ecr.sh --profile af-sandbox --region <region> \ --images-tar agent-fleet-images-<v>.tar.gz- The first form is for images built on this machine by
deploy/release/build.sh, the second for pushing from the images tar made on another machine release-ecr.shcopiesaf-engine-toolsinto ECR only when a60-enginesstack already exists (an upgrade). For the first60-engines, copy it yourself in Deploy the optional stacks belowaf-engine-toolsexists only in GHCR and is never built locally or shipped, so it is copied registry to registry withcrane copyrather than tagged and pushed. If neither ECR nor GHCR has that tag, the script stops. Runengine-tools-image.ymlwith that tag first
Push by hand
Section titled “Push by hand”The script is a transcription of these commands.
aws ecr get-login-password --region <region> | docker login --username AWS \ --password-stdin <acct>.dkr.ecr.<region>.amazonaws.com# tag + push the images already built for local (no heavy rebuild here):docker tag agent-fleet/control-plane:<tag> <acct>.dkr.ecr.<region>.amazonaws.com/af-control-plane:<tag>docker tag agent-fleet/workspace:<tag> <acct>.dkr.ecr.<region>.amazonaws.com/af-workspace:<tag>docker push <acct>.dkr.ecr.<region>.amazonaws.com/af-control-plane:<tag>docker push <acct>.dkr.ecr.<region>.amazonaws.com/af-workspace:<tag>Deploy the optional stacks
Section titled “Deploy the optional stacks”After the images are pushed and before 30-ingress, deploy only what you use, in this order: 40-ec2-pool (ecs-ec2), 50-tts (speech), 60-engines (self-hosted inference).
-
The
50-ttsand60-enginesservices cannot start without their engine images in ECR (af-voicevox;af-engine-tools,af-llamacpp,af-comfyui), and the stack then sits inCREATE_IN_PROGRESS. Put them in ECR before the stack -
For the first
60-engines, copyaf-engine-toolsfrom GHCR under60-engines’EngineToolsImageTagTerminal window crane copy ghcr.io/k-k1/agent-fleet/engine-tools:<EngineToolsImageTag> <acct>.dkr.ecr.<region>.amazonaws.com/af-engine-tools:<EngineToolsImageTag> -
standup.shcopies the images in this order before deploying, but only when rebuilding from saved parameters. For a first build, copy them by hand:af-voicevoxfor speech is under “Turn it on” in Add Japanese read-aloud, andaf-llamacpp/af-comfyuifor the inference engines are step 3 of “Turn it on” in Add inference engines -
Each stack’s parameters and images are in Start using the EC2 slot pool, Add Japanese read-aloud and Add inference engines
Start the Control Plane (30-ingress)
Section titled “Start the Control Plane (30-ingress)”The Control Plane does the sign-in itself (AuthMode=oauth): Google, or any OIDC IdP (Entra ID, Okta, Keycloak, Auth0, Cognito, GitLab). The ALB only terminates TLS.
The Control Plane keeps its state in RDS Postgres and workspace homes live on EFS. Both are wired from 10-data’s exports, so 10-data must already be up.
1. Put the Control Plane’s secrets in SSM
Section titled “1. Put the Control Plane’s secrets in SSM”The secrets live in SSM SecureString parameters and are read by the execution role. They never go into a plaintext environment variable.
RG=ap-northeast-1aws ssm put-parameter --profile af-sandbox --region $RG --type SecureString \ --name /af-cp/cookie-secret --value "$(openssl rand -hex 32)"aws ssm put-parameter --profile af-sandbox --region $RG --type SecureString \ --name /af-cp/master-key --value "$(openssl rand -hex 32)"# one of these two, or both — only the IdPs you enable are wired inaws ssm put-parameter --profile af-sandbox --region $RG --type SecureString \ --name /af-cp/google-client-secret --value "<GOOGLE_OAUTH_CLIENT_SECRET>" # your terminal, not sharedaws ssm put-parameter --profile af-sandbox --region $RG --type SecureString \ --name /af-cp/oidc-client-secret --value "<AF_OIDC_<ID>_CLIENT_SECRET>"- Put only the secrets of the IdPs you use. Type the values in your own terminal, never on a shared screen
- To encrypt
master-keywith your own KMS key instead ofaws/ssm, add--key-id <key ARN>to itsput-parameterand set20-platform’sMasterKeyKmsKeyArnto the same ARN first: the execution role’skms:Decryptmust exist before the Control Plane task starts. Details are in Put the keys at rest on AWS KMS
The GitHub and Bitbucket OAuth apps are not deployment configuration. There is no CloudFormation parameter and no SSM entry for them; a tenant administrator registers them in the Console (Choose the OAuth apps for integrations).
2. Register the redirect URI with your IdP
Section titled “2. Register the redirect URI with your IdP”Add https://<your-fqdn>/oauth2/callback to your IdP client’s authorized redirect URIs. That one URI serves every IdP you enable.
How to create the client for each IdP is in Set up how people sign in.
3. Deploy 30-ingress
Section titled “3. Deploy 30-ingress”30-ingress.yaml sits just under the 51,200-byte limit and is allowed to cross it. So that a future release that does cross it does not stop, always hand it over through the S3 bucket 20-platform created (Keep templates under the size limit).
For a deployment that signs in with Google:
CFN_BUCKET=$(aws cloudformation describe-stacks --stack-name <platform stack> --query "Stacks[0].Outputs[?OutputKey=='CfnTemplatesBucket'].OutputValue" --output text --profile <p> --region <r>)aws cloudformation deploy --stack-name af-ecs-ingress \ --template-file cfn/30-ingress.yaml --s3-bucket "$CFN_BUCKET" --s3-prefix cfn --capabilities CAPABILITY_NAMED_IAM \ --parameter-overrides GoogleClientId=<id> \ Fqdn=af.example.com HostedZoneId=<your-zone-id> \ AllowedEmails=you@example.com SuperAdminEmails=you@example.com \ --profile af-sandbox --region $RGFor a Microsoft Entra ID deployment (or Okta, Keycloak and the like), pass the Oidc* parameters instead of GoogleClientId. With GoogleClientId empty, Google is simply not offered.
--parameter-overrides OidcProviderId=ENTRA \ OidcIssuer=https://login.microsoftonline.com/<tenant-guid>/v2.0 \ OidcClientId=<application-client-id> OidcTrust=issuer \ OidcLabelJa="Microsoft でサインイン" OidcLabelEn="Sign in with Microsoft" \ Fqdn=af.example.com HostedZoneId=<your-zone-id> \ AllowedDomains=example.co.jp SuperAdminEmails=you@example.co.jp| Parameter | Details |
|---|---|
Fqdn / HostedZoneId |
The Console’s name and the Route 53 hosted zone that contains it. The zone must be in this account |
AllowedEmails / AllowedDomains |
The email addresses and domains allowed to sign in |
SuperAdminEmails |
Who becomes a deployment administrator (super_admin) |
OidcIssuer |
For Entra ID, pin it to your tenant GUID. With /common/ or /organizations/ every Microsoft account in the world reaches the sign-in, and personal accounts can rewrite their own email, so the Control Plane refuses to start unless OidcAllowedTids is set |
ImageTag |
For release images, add ImageTag=<v>. The default dev matches a :dev push for development |
ACM’s DNS validation and the Route 53 alias are automated. Certificate issuance adds a few minutes to the first deploy.
4. Check that it works
Section titled “4. Check that it works”- Check that
https://<your-fqdn>/readyzanswers 200./healthzanswers 200 even with a dead database, so/readyzis the one to check - Open the Console and sign in as one of the
SuperAdminEmails - If you split spend per member, start one workspace now (Split spend per member)
Split spend per member (optional)
Section titled “Split spend per member (optional)”Optional, but it cannot be backfilled: if you want it, start on the day you build.
Deploying the stacks does none of this. The templates carry only the IAM permission (ce:GetCostAndUsage on CpTaskRole). The tags are written by the running Control Plane, and there is no CloudFormation resource that activates a cost allocation tag. The order is:
- Before the build, turn on billing access (Turn on billing access)
- Deploy the stacks and start the Control Plane (the steps above on this page)
- Start one workspace. The Control Plane writes the tags onto resources. Start it on the day you build, even if nobody uses it yet
- Wait for AWS to discover the tag keys (up to 24 hours)
- Activate the cost allocation tags. Normally the Control Plane does this for you (see below)
| Point | Details |
|---|---|
| Not retroactive | Spend from before activation can never be attributed to anyone. Neither can the spend during the wait |
| Once per key per account | Activation is keyed on the tag key alone, with no value dimension. New tenants and members need nothing |
| Payer account only | Under AWS Organizations only the management (payer) account can activate |
| An unseen key cannot be activated | A key AWS has never seen on a real resource is refused with ValidationException: Tag keys not found |
Do not activate af-workspace |
Its value is derived from the member’s email address, and activating it copies that into the billing data (CUR, Cost Explorer, invoice CSVs). The Control Plane joins on af-membership, an opaque random id |
The Control Plane retries activation on every poller tick until AWS has discovered each key. It skips af-workspace and leaves alone any tag a person switched off.
Activate by hand only if you removed the CostAllocationTagActivation statement from CpTaskRole, or if the Console reports it could not.
aws ce update-cost-allocation-tags-status --region us-east-1 \ --cost-allocation-tags-status '[{"TagKey":"af-membership","Status":"Active"}, {"TagKey":"af-tenant","Status":"Active"},{"TagKey":"af-role","Status":"Active"}, {"TagKey":"af-pool","Status":"Active"},{"TagKey":"af-slot-size","Status":"Active"}]'list-cost-allocation-tags shows what AWS has found. A key listed as Inactive can be made Active at once; a key missing from the list is one nothing has written yet. The wait applies only the first time a key appears.
Related
Section titled “Related”For agents
Send this prompt to your agent to read this page.
Read https://agent-fleet.org/docs/operate/aws-ecs/install/index.md