Build on GKE
Build the cluster and everything around it with Terraform, apply the manifests, and get to the point where the Console answers on https://<fqdn>.
The preconditions the cluster must meet (P1–P15) are in The Kubernetes deployment at a glance; on GKE, Terraform meets all of them.
How it works
Section titled “How it works”| What you do | What happens |
|---|---|
| 1. Sign in to the project, enable the APIs, prepare a state bucket | Terraform can run |
2. terraform apply |
The VPC, the cluster and its two node pools, Cloud SQL, the KMS key, the load balancer’s address, certificate and DNS, the StorageClass and the IAM grants are created |
| 3. Point kubectl at this cluster | Every later kubectl reaches this cluster and no other |
| 4. Grant the database (once) | The CP’s IAM database user owns the database |
| 5. Copy and edit the overlay | The deployment’s own settings (URL, sign-in, image version) are set |
| 6. Apply the overlay and create the Secret | The CP starts |
| 7. Know the load balancer | You know when a WebSocket is cut |
| 8. Check it | You know the rights, the isolation, name resolution and the home’s owner are right |
Shell variables
Section titled “Shell variables”LOCATION is where the cluster itself is: the region for a regional cluster, the zone (node_zones[0]) for a zonal one such as the small profile.
Every command below runs from the repository root, and every gcloud command names the project explicitly, so a different default project in your gcloud configuration cannot be the one that is changed. Set these once per shell:
PROJECT=<project> # Terraform's project_idREGION=<region> # Terraform's regionLOCATION="$REGION" # the cluster's location: node_zones[0] instead when zonal_cluster = truePREFIX=<name_prefix> # Terraform's name_prefixTF="terraform -chdir=deploy/gcp/gke"OVERLAY="deploy/kubernetes/overlays/$PREFIX" # your overlay; point it at the copy once you move it1. Prepare the project (once)
Section titled “1. Prepare the project (once)”- Sign in twice:
gcloudfor the commands below, and Application Default Credentials for Terraform’s providersTerminal window gcloud auth logingcloud auth application-default logingcloud auth application-default set-quota-project "$PROJECT" - Enable the APIs Terraform uses
Terminal window gcloud services enable container.googleapis.com compute.googleapis.com \sqladmin.googleapis.com servicenetworking.googleapis.com certificatemanager.googleapis.com \dns.googleapis.com cloudkms.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \--project "$PROJECT" - Check the regional SSD quota against what the deployment can reach, and raise it now if it falls short (Disk quota (GKE)): a fresh project’s 500 GB does not hold a second workspace node with the defaults
- Check that the Cloud DNS managed zone for your domain already exists in the project (P13)
- Create the state bucket if you do not have one, with versioning on
Terminal window gcloud storage buckets create gs://<state-bucket> --project "$PROJECT" --location "$REGION" \--uniform-bucket-level-accessgcloud storage buckets update gs://<state-bucket> --versioning --project "$PROJECT" - Decide where the images come from
| Source | Setting |
|---|---|
| Your own Artifact Registry repository | Push control-plane and workspace and set artifact_registry_repository, so the node pool and the CP get reader on it |
The public registry (ghcr.io/k-k1/agent-fleet/…) |
Leave artifact_registry_repository null |
2. Apply Terraform
Section titled “2. Apply Terraform”cp deploy/gcp/gke/terraform.tfvars.example deploy/gcp/gke/terraform.tfvars # fill it in; gitignored$TF init -backend-config="bucket=<state-bucket>" -backend-config="prefix=$PREFIX"$TF applyThe variables in terraform.tfvars to watch:
| Variable | What to watch |
|---|---|
name_prefix |
Names every resource and the two namespaces (<prefix>-cp, <prefix>-ws). Two deployments in one project need two prefixes (P8) |
authorized_networks |
Who may reach the control-plane endpoint. The provider that creates the StorageClass talks to that endpoint, so apply must run from an authorised network — or, with enable_private_endpoint = true, from inside the VPC |
node_cidr, pod_cidr, service_cidr, control_plane_cidr, private_service_access_cidr |
Cannot change after the cluster exists. The first four must lie inside RFC 1918 or 100.64.0.0/10, and a plan fails otherwise (P15). Keep the Private Service Access range there too: the workspace egress policy allows every address outside those ranges and link-local 169.254.0.0/16 (the metadata server included), so it blocks this range only when the range lies inside them, and the plan does not check this one |
min_master_version |
A floor, not the version the cluster is created at: the cluster starts at the release channel’s default, and every plan fails if the control plane runs below the floor (P1) |
The certificate is issued once the DNS authorisation records resolve. This shows its state; it is usually ACTIVE within an hour:
gcloud certificate-manager certificates describe $PREFIX-cert --project "$PROJECT"What Terraform grants
Section titled “What Terraform grants”Each grant goes to one principal on one resource. Nothing names the workspace namespace.
| Principal | What it gets |
|---|---|
| The CP’s service account | Cloud SQL client and instance user, conditioned to this instance, and Artifact Registry reader on the repository |
| The node service account | Artifact Registry reader and the minimum node role (roles/container.defaultNodeServiceAccount: logs, metrics, and from 1.33 the autoscaler’s metrics) |
| GKE’s service agent | Encrypt/decrypt on the Secrets key |
The CP’s Kubernetes service account <prefix>-cp/af-cp |
May act as the CP’s service account |
A small profile for a trial
Section titled “A small profile for a trial”For a trial or a single team. The commented block at the end of terraform.tfvars.example sets all of it:
| Setting | Value | The defaults instead |
|---|---|---|
node_zones, zonal_cluster |
one zone, true |
a regional cluster with nodes in two zones |
sql_availability_type, sql_tier |
ZONAL, db-custom-1-3840 (the smallest dedicated-core tier Postgres offers) |
REGIONAL |
system_machine_type, system_node_count |
e2-standard-2, 1 |
e2-standard-2 in each of two zones |
workspace_machine_type |
n2-standard-4 |
n2-standard-8 |
managed_prometheus |
false |
on |
What it gives up:
- Control-plane high availability. A zonal cluster’s API server is unavailable during its upgrades, when running workspaces keep running but cannot be managed
- Database high availability. Cloud SQL
ZONALhas no standby, so a zone outage or a maintenance restart takes the CP down until the instance returns. Backups and point-in-time recovery stay on - Zone-failure tolerance of every kind. The CP, its disk, the workspaces’ volumes and nodes are all in one zone; in a zone outage running workspaces stop with it
zonal_cluster replaces the cluster, so decide before the first apply; moving an existing deployment is a rebuild.
- A workspace pod’s limits must fit the
n2-standard-4’s ~4 vCPU / 16 GiB - With a single e2-medium system node (about 0.94 vCPU allocatable) the CP stays
Pending(“Insufficient cpu”) and its claim staysPendingwith it. That is why the profile usese2-standard-2. Changingsystem_machine_typeon a running cluster is an in-place node pool update of about 6 minutes - The profile has been stood up once, on a zonal cluster (the first
applytook about 18 minutes, and the CP was healthy about 14 minutes after the overlay was applied). Other zones, a regional variant and other regions are untried, and so are shared-core Cloud SQL tiers (db-g1-small)
The cost estimate is in Pause and see the bill.
3. Point kubectl at this cluster
Section titled “3. Point kubectl at this cluster”Terraform does not touch your kubeconfig. Before any kubectl command, fetch this cluster’s credentials and check that the current context is it — otherwise the namespaces, RBAC and Secrets below land on whatever cluster your kubeconfig pointed at:
eval "$($TF output -raw get_credentials)" # gcloud container clusters get-credentials …kubectl config current-context # gke_<project>_<region>_<prefix>-gkekubectl get nodes -L agent-fleet.io/pool # the system and workspace poolsDo the same in every new shell before the procedures on the operating pages.
4. Grant the database (once)
Section titled “4. Grant the database (once)”The CP’s database user is an IAM user with no password and, at first, no rights. Give it the database, as the built-in postgres user, from a temporary pod inside the cluster.
- The instance has a private IP only, a workspace is outside the VPC, and
cloud-sql-proxyis not installed there, so the grant runs from a pod on the cluster’s nodes (a short-livedpostgres:17-alpinepod with a restricted security context) - Keep the
postgrespassword in your vault, not here
With a terminal
Section titled “With a terminal”gcloud sql users set-password postgres --instance "$PREFIX-pg" --project "$PROJECT" --prompt-for-passwordDBUSER="$($TF output -raw cp_database_user)"DBHOST="$(gcloud sql instances describe "$PREFIX-pg" --project "$PROJECT" --format='value(ipAddresses[0].ipAddress)')"# The statements to paste into psql below (the pod knows nothing of $DBUSER):echo "GRANT \"$DBUSER\" TO postgres; ALTER DATABASE agentfleet OWNER TO \"$DBUSER\"; ALTER SCHEMA public OWNER TO \"$DBUSER\";"# The CP namespace does not exist yet (step 6 creates it); `default` does.kubectl -n default run psql-grant -it --rm --restart=Never --image=postgres:17-alpine \ --override-type=strategic \ --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":70,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"psql-grant","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' \ -- psql "host=$DBHOST user=postgres dbname=agentfleet sslmode=require" -W- At the
Password:prompt type thepostgrespassword - Paste the printed statements at the
psqlprompt. ExpectGRANT ROLE,ALTER DATABASEandALTER SCHEMAback; anyERRORmeans the grant is not done - Leave with
\q
- Interactive
psqldoes not exit on a SQL error, so read the replies - The password goes to
psqlover the pod’s terminal only: it is in no command line, pod spec or API object - The pod is removed on exit (
--rm); if the session dropped, delete it withkubectl -n default delete pod psql-grant
Without a terminal (a script, CI, an agent’s shell)
Section titled “Without a terminal (a script, CI, an agent’s shell)”-it and -W need a terminal, so the command above cannot run there. Instead, store the password in a temporary Secret, run a non-interactive pod that takes PGPASSWORD from it and the statements from -c, then delete both. The password stays out of argv and out of the pod spec.
The first line of the block above (gcloud sql users set-password … --prompt-for-password) needs a terminal too: without one, set the password with --password (it is in that one gcloud call’s argument list for a moment), and set a new random value afterwards. For example, set a generated one first; the block below then uses PGPW instead of prompting:
PGPW="$(openssl rand -base64 24)" # kept only in this shellgcloud sql users set-password postgres --instance "$PREFIX-pg" --project "$PROJECT" --password "$PGPW"The grant itself defines DBUSER and DBHOST on its own and stops if either is empty:
DBUSER="$($TF output -raw cp_database_user)"DBHOST="$(gcloud sql instances describe "$PREFIX-pg" --project "$PROJECT" --format='value(ipAddresses[0].ipAddress)')"[ -n "$DBUSER" ] && [ -n "$DBHOST" ] || { echo "DBUSER or DBHOST is empty: stop"; exit 1; }PWDIR="$(mktemp -d)"; chmod 700 "$PWDIR"cleanup() { rm -rf "$PWDIR" # the local copy first: it needs no API kubectl -n default delete pod psql-grant --ignore-not-found --wait=false --request-timeout=30s \ || echo "could not delete pod psql-grant: delete it by hand" kubectl -n default delete secret psql-grant-pw --ignore-not-found --request-timeout=30s \ || echo "WARNING: secret psql-grant-pw may remain (it holds the postgres password): delete it by hand"}trap cleanup EXIT # runs under `set -e` too[ -n "${PGPW:-}" ] || { read -rs -p 'postgres password: ' PGPW; echo; } # or read it from your vaultprintf '%s' "$PGPW" > "$PWDIR/PGPASSWORD"; unset PGPW # printf is a builtin: not in any argvkubectl -n default create secret generic psql-grant-pw --from-file=PGPASSWORD="$PWDIR/PGPASSWORD"kubectl -n default run psql-grant --restart=Never --image=postgres:17-alpine \ --override-type=strategic \ --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":70,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"psql-grant","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}},"env":[{"name":"PGPASSWORD","valueFrom":{"secretKeyRef":{"name":"psql-grant-pw","key":"PGPASSWORD"}}}]}]}}' \ -- psql -v ON_ERROR_STOP=1 "host=$DBHOST user=postgres dbname=agentfleet sslmode=require" \ -c "GRANT \"$DBUSER\" TO postgres; ALTER DATABASE agentfleet OWNER TO \"$DBUSER\"; ALTER SCHEMA public OWNER TO \"$DBUSER\";"kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded pod/psql-grant --timeout=120s \ || echo "grant pod did not succeed"kubectl -n default logs psql-grant # expect GRANT ROLE, ALTER DATABASE, ALTER SCHEMA- The
EXITtrap removes the password file first, then tries each delete on its own, so one failingkubectl delete(expired login, no connection) does not skip the rest; if the Secret could not be deleted it says so and you delete it by hand - The trap runs even when
waittimes out or a command fails underset -e; the|| echokeepslogsreachable so theERRORcan be read before the trap fires - The password file sits in a mode-700 directory on your machine for the length of the run; use a FIFO or your vault’s own file mechanism if that is not acceptable
- If the pod ends
Failed,logsshows theERRORand the grant is not done - Afterwards set a new
postgrespassword (gcloud sql users set-password … --prompt-for-password, or a random value you do not keep): the old one was in a Secret object for the length of the run
Only the variant without a terminal has been run as written (with the read line replaced by a generated password). The interactive -it … -W form has not been run exactly as written.
5. Prepare the overlay
Section titled “5. Prepare the overlay”cp -r deploy/kubernetes/overlays/gke "$OVERLAY"$TF output -raw kustomize_deployment > "$OVERLAY/deployment.yaml"Then edit, in the copy:
| File | What to change |
|---|---|
cp.env |
PUBLIC_BASE_URL and PUBLIC_DOMAIN (the fqdn), AF_PREVIEW_DOMAIN (Terraform’s preview_domain, or empty), the sign-in settings, SUPER_ADMIN_EMAILS. Keep AF_TRUSTED_PROXY_HOPS=2 (7. Know the load balancer) |
kustomization.yaml |
images: — the CP image and the release tag |
base/cp-data.yaml |
Asks 20 GiB for the CP’s own disk; patch it in the copy if the internal git provider will hold more |
Keep it outside the clone
Section titled “Keep it outside the clone”Keep this copy in your own repository or vault, not in a working copy of this one. Moved out of the clone, two things in kustomization.yaml must change:
- The paths.
resources: ../../baseandcomponents: ../../components/gkeand../../components/paramsare relative todeploy/kubernetes/overlays/<name>/inside this repository and resolve to nothing elsewhere. Point them at the clone’sdeploy/kubernetes/baseanddeploy/kubernetes/components/…with relative paths from your overlay:kubectl kustomizerejects an absolute directory reference (new root '…' cannot be absolute). Check withkubectl kustomize <your overlay>before applying OVERLAY. Set it to the copy’s path (OVERLAY=<your overlay>) in every shell; every command on these pages that reads or applies the overlay uses$OVERLAY, so none of them touches the clone’s copynewName. The copiedimages:entry carries a placeholdernewName(an Artifact Registry path). With the public images (ghcr.io/k-k1/agent-fleet/…) delete thenewNameline and keepnewTag, so the name stays the public one; with your own Artifact Registry repository setnewNameto it
Archive egress events (optional)
Section titled “Archive egress events (optional)”The CP can write each past UTC day of raw egress events to Cloud Storage before it purges them (AF_EGRESS_ARCHIVE_URL; Control outbound traffic). It is off by default: with egress_archive = false the plan has no bucket, no grant and no new setting for the CP.
| Variable | Default | Meaning |
|---|---|---|
egress_archive |
false |
true turns it on |
egress_archive_existing_bucket |
"" |
Name of a bucket you already have. The module then manages only the CP’s grants on it. Empty: the module creates <name_prefix>-egress-archive-<project_id> |
egress_archive_prefix |
"" |
Object prefix inside the bucket (the CP already writes under egress/). Segments of A-Za-z0-9._- joined by single slashes (team/prod); no ./.. segment, no #?%, no slash at either end |
egress_archive_location |
"" |
Location of a created bucket; empty is region |
egress_archive_retention_days |
null |
Delete archived objects after this many days (created bucket only). null never deletes: the archive is the only copy once the events are purged |
- A created bucket has uniform bucket-level access and public access prevention enforced, and
force_destroy = false: Destroy stops at a bucket that still holds archived days - The CP’s existing Google service account (the Workload Identity one; no second identity) gets
roles/storage.objectCreatorandroles/storage.objectVieweron that bucket only - The viewer role is not optional: the next run reads back a day it already wrote to verify it, and without it that read is a 403 and the purge stops at that day
- A bucket you name is not modified apart from those two members; keep its public access prevention on yourself
cp.env is touched only after terraform output succeeded: drop the old key and add the current line only when there is one. An empty output means the archive is off, and a key left behind would keep the CP archiving to a bucket it no longer has grants on.
if line=$($TF output -raw egress_archive_env); then sed -i '/^AF_EGRESS_ARCHIVE_URL=/d' "$OVERLAY/cp.env" [ -z "$line" ] || printf '%s\n' "$line" >> "$OVERLAY/cp.env"else echo "terraform output failed: cp.env left as it was" >&2; falsefi- A failed read (backend error, wrong
-chdir, output not created yet) is never taken for “archive off”: that would delete the key and make the next apply purge events unarchived. The block exits non-zero on a failed read: do not re-apply then - Then re-apply the overlay.
kustomizerefuses acp.envthat repeats the key, so do not append twice - Not yet verified against a real bucket: the first archive, a second run that finds and verifies the objects, and the 403 when the viewer role is missing. The offline tests check the plan only
6. Apply the overlay and create the Secret
Section titled “6. Apply the overlay and create the Secret”The CP reads one Secret, af-cp-secrets, which these manifests never contain. Create it once the namespace exists:
kubectl config current-context # still this cluster (step 3)kubectl apply -k "$OVERLAY"kubectl -n "$PREFIX-cp" create secret generic af-cp-secrets \ --from-literal=AF_MASTER_KEY="$(cat master-key)" \ --from-literal=AF_COOKIE_SECRET="$(openssl rand -hex 32)" \ --from-literal=GOOGLE_OAUTH_CLIENT_SECRET="$(cat google-client-secret)"- Until it exists the CP pod waits in
CreateContainerConfigError, and starts by itself once it does - Generate
AF_MASTER_KEYwithhead -c 32 /dev/urandom | base64, as for compose, and keep it somewhere durable that outlives this cluster (Secret Manager or your vault), not only in the Kubernetes Secret or a file on your machine (P14) - It is needed again whenever the CP is recreated — a new cluster, or
af-cp-secretsdeleted — and a different key cannot read what the old one encrypted - Other sign-in secrets (
AF_OIDC_<ID>_CLIENT_SECRET,GITHUB_OAUTH_CLIENT_SECRET) go in the same Secret - Rotate one with
kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -and thenkubectl -n "$PREFIX-cp" rollout restart deployment/af-cp
What the keys must be is in Handle the deployment keys.
7. Know the load balancer
Section titled “7. Know the load balancer”The Gateway af-cp (components/gke/gateway.yaml) makes GKE build a global external Application Load Balancer — not the classic one, which closes even an active WebSocket at the backend timeout. Terminals, the mirror and the browser pane are WebSockets, so two limits matter:
| WebSocket | When it is cut |
|---|---|
| Idle (a terminal nobody types in, a quiet session) | After the backend service’s timeoutSec. The default is 30 seconds; the GCPBackendPolicy af-cp in the same file sets 3600 |
| Active | After 24 hours whatever timeoutSec says. That cut cannot be configured away; the Console has to reconnect |
- To change
timeoutSec, edit it in that file andkubectl apply -kthe overlay; GKE updates the backend service within a few minutes (gcloud compute backend-services list --project "$PROJECT"shows the value) - The Console pings on the same socket, a round-trip ping every 5 seconds and a heartbeat every 15. A terminal nobody typed in has stayed connected for more than 67 minutes past the 3600-second
timeoutSec, because the pings kept the connection from looking idle - A browser may slow or stop those timers for a hidden tab, a frozen page or a sleeping machine; what happens then has not been checked
- The 24-hour cut has not been waited out, but a forced cut (a CP restart) has: the terminal pane shows
[disconnected], and clicking it reattaches within a few seconds with the shell and its tmux session intact, without reloading the page. A cut by the load balancer reaches the browser as the same unclean close, so the same recovery applies
AF_TRUSTED_PROXY_HOPS stays 2
Section titled “AF_TRUSTED_PROXY_HOPS stays 2”The load balancer appends <client>, <load balancer> to whatever X-Forwarded-For it receives, and the CP counts from the right. With nothing in front of the load balancer, 2 names the real client.
- Raising it for a CDN or proxy in front is only safe when the load balancer cannot be reached except through that proxy (for example a Cloud Armor policy admitting only the proxy’s addresses) and you have checked which entries the proxy appends or replaces
- Otherwise a client that sends its own
X-Forwarded-Forstraight to the load balancer chooses the address the CP sees, and walks past a tenant’s network restriction - This setup supports no proxy in front of the load balancer
8. Check it
Section titled “8. Check it”kubectl -n "$PREFIX-cp" rollout status deployment/af-cpkubectl get ns "$PREFIX-ws" -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}' # restrictedkubectl -n "$PREFIX-cp" get pvc af-cp-data # Boundkubectl -n "$PREFIX-cp" get gateway af-cp # PROGRAMMED True, ADDRESS = lb_addresscurl -sS https://<fqdn>/readyzPoll /readyz itself; PROGRAMMED True is not enough. A rollout that is done does not mean the load balancer is:
- It has taken about 5 to 6 more minutes after the CP was ready before
/readyzreturned 200, andcurlprinted000(no connection) in the meantime, although DNS already pointed at the new address - The Gateway has turned
PROGRAMMED Trueand/readyzstill answered000for about 4 minutes after that
Retry the request until it answers:
until [ "$(curl -sS -m 10 -o /dev/null -w '%{http_code}' https://<fqdn>/readyz)" = 200 ]; do sleep 30; donekubectl -n "$PREFIX-cp" get gateway af-cp -w shows the Gateway’s own state. A kubectl wait --for=condition=Programmed gateway/af-cp would stop too early for the same reason (it has not been tried).
What the CP’s account may do
Section titled “What the CP’s account may do”Every line must answer as shown:
SA=system:serviceaccount:$PREFIX-cp:af-cpkubectl auth can-i create statefulsets -n "$PREFIX-ws" --as $SA # yeskubectl auth can-i delete pods -n "$PREFIX-ws" --as $SA # yes (the erase pod)kubectl auth can-i patch namespaces --as $SA # nokubectl auth can-i create networkpolicies -n "$PREFIX-ws" --as $SA # nokubectl auth can-i list storageclasses --as $SA # nokubectl auth can-i get "storageclass/$PREFIX-workspace" --as $SA # yeskubectl auth can-i create pods -n "$PREFIX-cp" --as $SA # noNodeLocal DNSCache is off
Section titled “NodeLocal DNSCache is off”That is P11; this prints nothing:
kubectl -n kube-system get pods -l k8s-app=node-local-dnsWhere the workspace runs, and name resolution
Section titled “Where the workspace runs, and name resolution”Sign in, start a workspace, and check that its pod runs on the workspace pool and resolves names:
kubectl -n "$PREFIX-ws" get pods -o wide -l agent-fleet.io/workspacekubectl -n "$PREFIX-ws" exec <pod> -- getent hosts github.com # an address, not emptyNo node answers a workspace
Section titled “No node answers a workspace”A workspace must reach no node, its own node included (P15). The pod itself reports open or closed.
- Every node line must say
closed, and the last line, the control,open - An
ERRORline, or no node lines at all, means nothing was checked
nodes=$(kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}') \ && [ -n "$nodes" ] || echo "ERROR: could not list the nodes"probe() { # <ip> <port> kubectl -n "$PREFIX-ws" exec <pod> -- bash -c \ "if timeout 3 bash -c 'exec 3<>/dev/tcp/$1/$2' 2>/dev/null; then echo '$1:$2 open'; else echo '$1:$2 closed'; fi" \ || echo "$1:$2 ERROR: exec failed"}for ip in $nodes; do for port in 22 10250 10256; do probe "$ip" "$port"; done; doneprobe 1.1.1.1 443The home’s owner
Section titled “The home’s owner”In that workspace’s terminal, the home has to be dev’s and writable by nobody else, or af-gcloud-exec and af-aws-exec refuse to keep their state under it:
stat -c '%U:%G %A' /home/dev # dev:dev drwxr-sr-x — no w for group or other (the s, from fsGroup, is harmless)- The home is the directory
.af-homeof the home claim, which an init container running asdevcreates — and moves an earlier home’s files into — before the agent starts - The claim’s root stays
root:devwith group write, as the kubelet leaves it
When it goes wrong
Section titled “When it goes wrong”| Symptom | Cause and what to do |
|---|---|
The CP pod stays in CreateContainerConfigError |
af-cp-secrets does not exist yet. Create it in 6. and the pod starts by itself |
curl https://<fqdn>/readyz prints 000 |
The load balancer is not ready yet. It takes minutes; wait with the loop in 8. |
The certificate does not turn ACTIVE |
The DNS authorisation records do not resolve yet; usually within an hour |
A workspace pod stays Pending and the scale-up says QUOTA_EXCEEDED |
The SSD quota is used up. Disk quota (GKE) |
A workspace pod is stuck in Init |
kubectl logs <pod> -c home-layout names the reason. After a rollback, see Roll back past the home layout |
| Every lookup from a workspace fails | NodeLocal DNSCache is on (P11) |
kubectl kustomize says new root '…' cannot be absolute |
The overlay’s paths are absolute. Keep it outside the clone |
Related
Section titled “Related”- The Kubernetes deployment at a glance
- Back up and restore: give the CP’s disk a snapshot schedule after the first start
- Set up alerts
- Set up how people sign in
For agents
Send this prompt to your agent to read this page.
Read https://agent-fleet.org/docs/operate/kubernetes/gke/index.md