Build on another cluster
Stand Agent Fleet up on a cluster other than GKE from the generic overlay, deploy/kubernetes/overlays/generic.
That overlay is the base with nothing provider-specific. Everything Terraform does on GKE is yours to provide.
How it works
Section titled “How it works”| What you do | What happens |
|---|---|
| Meet preconditions P1–P14 yourself and check that the ranges meet P15 | The cluster can isolate workspaces. Nothing else checks them |
Copy, edit and apply overlays/generic, then create the Secret with AF_DATABASE_URL |
The namespaces, the CP, RBAC and NetworkPolicies are created, and the CP starts |
Put your own ingress in front of the Service af-cp |
The Console is reachable from outside |
Add an ingress policy in the CP namespace before the first apply, and a backup for af-cp-data |
The same protection as the GKE setup, from the CP’s first start |
The list of preconditions is in The Kubernetes deployment at a glance.
Set the shell variables and copy the overlay. If it lives outside this clone, fix its paths as in Keep it outside the clone.
OVERLAY=<your overlay> # where your copy of overlays/generic livesCP_NS=<cpNamespace> # deployment.yaml's cpNamespaceWS_NS=<wsNamespace> # deployment.yaml's wsNamespacecp -r deploy/kubernetes/overlays/generic "$OVERLAY"- Set the copy’s
deployment.yamlfrom your cluster- the namespaces
cpNamespaceandwsNamespace(with your own prefix) - a StorageClass that meets P3
workspaceImage, and the CP’s tag underimages:inkustomization.yaml- check too that the pod, service, node and control-plane ranges meet P15
- the namespaces
- Edit
cp.env:PUBLIC_BASE_URL,PUBLIC_DOMAIN, the sign-in settings andSUPER_ADMIN_EMAILS, and where needed:
| When | Setting |
|---|---|
| Workspaces must run on particular nodes | AF_K8S_NODE_SELECTOR in cp.env |
| The images need a pull secret | AF_K8S_IMAGE_PULL_SECRET in cp.env; the secret itself is created in step 5 |
- Restrict the CP’s ingress before anything starts. The base has no ingress policy for the CP namespace, and anything that reaches port 8099 can name any user. Copy
components/gke/cp-networkpolicy.yamlinto the overlay and edit it:- replace the two Google
ipBlockentries of the 8099 rule with the source of your ingress (its controller pods’ namespace and labels, or its addresses) - in the 8098 rule, replace
af-placeholder-wswith yourwsNamespace(the overlay does not rewrite that label; it does rewrite the policy’s own namespace)
Terminal window cp deploy/kubernetes/components/gke/cp-networkpolicy.yaml "$OVERLAY/cp-networkpolicy.yaml"# edit it as above, add `- cp-networkpolicy.yaml` under resources: in kustomization.yaml, then:kubectl kustomize "$OVERLAY" | grep -A3 'name: cp-ingress' # namespace: your cpNamespace - replace the two Google
- Apply the overlay. It creates the two namespaces, the CP, RBAC and the NetworkPolicies, the CP’s ingress policy included; the CP pod waits in
CreateContainerConfigErroruntil step 5Terminal window kubectl config current-context # the cluster you meankubectl apply -k "$OVERLAY" - Create the CP’s Secret in the CP namespace, with the store’s DSN as
AF_DATABASE_URLbesides the keys, and wait for the rollout. Put each sign-in secret you use in the same Secret (GOOGLE_OAUTH_CLIENT_SECRET,AF_OIDC_<ID>_CLIENT_SECRET,GITHUB_OAUTH_CLIENT_SECRET); what the keys must be is in Handle the deployment keys. If the images need a pull secret, create it now in$WS_NSunder the name you gaveAF_K8S_IMAGE_PULL_SECRETTerminal window kubectl -n "$CP_NS" create secret generic af-cp-secrets \--from-literal=AF_MASTER_KEY="$(cat master-key)" \--from-literal=AF_COOKIE_SECRET="$(openssl rand -hex 32)" \--from-literal=AF_DATABASE_URL="$(cat database-url)" \--from-literal=GOOGLE_OAUTH_CLIENT_SECRET="$(cat google-client-secret)"kubectl -n "$CP_NS" rollout status deployment/af-cp - Put your cluster’s ingress in front of the Service
af-cp(port 8099), as the source the policy of step 3 admits- with WebSockets and a long idle timeout
- set
AF_TRUSTED_PROXY_HOPSto the number of proxies that append toX-Forwarded-For, counting a proxy only if the CP cannot be reached except through it (why: Build on GKE), and apply the overlay again
- Give the claim
af-cp-dataa backup. What it looks like on GKE is in Back up and restore - Meet P1–P14 yourself. Nothing else checks them
The rules for changing the manifests are in The Kubernetes deployment at a glance.
Related
Section titled “Related”- Build on GKE: its step 8 checks apply here too
- Replace nodes and recover from a stopped node
For agents
Send this prompt to your agent to read this page.
Read https://agent-fleet.org/docs/operate/kubernetes/other-clusters/index.md