Skip to content

Build on another cluster

Stand Agent Fleet up on a cluster other than GKE from the generic overlay, deploy/kubernetes/overlays/generic. That overlay is the base with nothing provider-specific. Everything Terraform does on GKE is yours to provide.

What you do What happens
Meet preconditions P1–P14 yourself and check that the ranges meet P15 The cluster can isolate workspaces. Nothing else checks them
Copy, edit and apply overlays/generic, then create the Secret with AF_DATABASE_URL The namespaces, the CP, RBAC and NetworkPolicies are created, and the CP starts
Put your own ingress in front of the Service af-cp The Console is reachable from outside
Add an ingress policy in the CP namespace before the first apply, and a backup for af-cp-data The same protection as the GKE setup, from the CP’s first start

The list of preconditions is in The Kubernetes deployment at a glance.

Set the shell variables and copy the overlay. If it lives outside this clone, fix its paths as in Keep it outside the clone.

Terminal window
OVERLAY=<your overlay> # where your copy of overlays/generic lives
CP_NS=<cpNamespace> # deployment.yaml's cpNamespace
WS_NS=<wsNamespace> # deployment.yaml's wsNamespace
cp -r deploy/kubernetes/overlays/generic "$OVERLAY"
  1. Set the copy’s deployment.yaml from your cluster
    • the namespaces cpNamespace and wsNamespace (with your own prefix)
    • a StorageClass that meets P3
    • workspaceImage, and the CP’s tag under images: in kustomization.yaml
    • check too that the pod, service, node and control-plane ranges meet P15
  2. Edit cp.env: PUBLIC_BASE_URL, PUBLIC_DOMAIN, the sign-in settings and SUPER_ADMIN_EMAILS, and where needed:
When Setting
Workspaces must run on particular nodes AF_K8S_NODE_SELECTOR in cp.env
The images need a pull secret AF_K8S_IMAGE_PULL_SECRET in cp.env; the secret itself is created in step 5
  1. Restrict the CP’s ingress before anything starts. The base has no ingress policy for the CP namespace, and anything that reaches port 8099 can name any user. Copy components/gke/cp-networkpolicy.yaml into the overlay and edit it:
    • replace the two Google ipBlock entries of the 8099 rule with the source of your ingress (its controller pods’ namespace and labels, or its addresses)
    • in the 8098 rule, replace af-placeholder-ws with your wsNamespace (the overlay does not rewrite that label; it does rewrite the policy’s own namespace)
    Terminal window
    cp deploy/kubernetes/components/gke/cp-networkpolicy.yaml "$OVERLAY/cp-networkpolicy.yaml"
    # edit it as above, add `- cp-networkpolicy.yaml` under resources: in kustomization.yaml, then:
    kubectl kustomize "$OVERLAY" | grep -A3 'name: cp-ingress' # namespace: your cpNamespace
  2. Apply the overlay. It creates the two namespaces, the CP, RBAC and the NetworkPolicies, the CP’s ingress policy included; the CP pod waits in CreateContainerConfigError until step 5
    Terminal window
    kubectl config current-context # the cluster you mean
    kubectl apply -k "$OVERLAY"
  3. Create the CP’s Secret in the CP namespace, with the store’s DSN as AF_DATABASE_URL besides the keys, and wait for the rollout. Put each sign-in secret you use in the same Secret (GOOGLE_OAUTH_CLIENT_SECRET, AF_OIDC_<ID>_CLIENT_SECRET, GITHUB_OAUTH_CLIENT_SECRET); what the keys must be is in Handle the deployment keys. If the images need a pull secret, create it now in $WS_NS under the name you gave AF_K8S_IMAGE_PULL_SECRET
    Terminal window
    kubectl -n "$CP_NS" create secret generic af-cp-secrets \
    --from-literal=AF_MASTER_KEY="$(cat master-key)" \
    --from-literal=AF_COOKIE_SECRET="$(openssl rand -hex 32)" \
    --from-literal=AF_DATABASE_URL="$(cat database-url)" \
    --from-literal=GOOGLE_OAUTH_CLIENT_SECRET="$(cat google-client-secret)"
    kubectl -n "$CP_NS" rollout status deployment/af-cp
  4. Put your cluster’s ingress in front of the Service af-cp (port 8099), as the source the policy of step 3 admits
    • with WebSockets and a long idle timeout
    • set AF_TRUSTED_PROXY_HOPS to the number of proxies that append to X-Forwarded-For, counting a proxy only if the CP cannot be reached except through it (why: Build on GKE), and apply the overlay again
  5. Give the claim af-cp-data a backup. What it looks like on GKE is in Back up and restore
  6. Meet P1–P14 yourself. Nothing else checks them

The rules for changing the manifests are in The Kubernetes deployment at a glance.

For agents

Send this prompt to your agent to read this page.

Read https://agent-fleet.org/docs/operate/kubernetes/other-clusters/index.md