Bootstrapping and maintaining a Google Cloud organization · Topic 5 of 6
Multiple Environments, GKE Fleets, Patching and Upgrades
Manage staging and production on Google Cloud: ephemeral environments, config and policy, GKE fleets, release channels and safe node upgrades.
- 5 min
- One project per environment; differences live in small overlays, not copies.
- Ephemeral environments are created and destroyed by the pipeline, and labelled.
- Fleets group clusters; Config Sync and Policy Controller keep them consistent.
What it is
Most services run in several environments: development, staging and production at least. The aim is that environments differ only where they must, such as size, data and access, so a change that works in staging behaves the same in production.
This page covers short-lived environments, keeping configuration and policy consistent, managing many GKE clusters as a fleet, and patching without outages.
How it works
Separating environments
Give each environment its own project, inside environment folders, with its own service accounts. Production then has separate IAM, quotas, budgets and network, and a test change cannot reach production resources.
Ephemeral environments
An ephemeral environment exists only for a pull request or a test run, then is deleted. It catches problems early without paying for idle copies.
| Platform | How to create one |
|---|---|
| Cloud Run | Deploy a revision with a tag and no traffic, which gets its own URL |
| GKE | A namespace per branch, deleted when the branch closes |
| Whole stack | Terraform or Infrastructure Manager with a unique name prefix, destroyed by the pipeline |
gcloud run deploy web --image="$IMAGE" --region=europe-west1 \
--tag=pr-142 --no-trafficLabel every ephemeral resource with its owner and pull request, and let the pipeline tear it down. A scheduled job that deletes anything older than its allowed lifetime catches the ones that were missed.
Managing configuration and policy
| Layer | Tool |
|---|---|
| Application config per environment | Kustomize overlays or Helm values, selected by Skaffold profiles in Cloud Deploy |
| Infrastructure per environment | The same Terraform modules with different variable files |
| Cluster config across many clusters | Config Sync from a Git repository |
| Kubernetes guardrails | Policy Controller constraints, such as required labels or no privileged pods |
| Cloud guardrails | Organization policies on each environment folder |
Keep one base and small overlays. Large copies of manifests per environment drift apart quickly.
GKE fleets
A fleet is a logical group of GKE clusters, and clusters registered from other environments, managed together. A fleet lives in a fleet host project.
| Fleet feature | What it gives |
|---|---|
| Config Sync | The same configuration applied to every cluster from Git |
| Policy Controller | The same policies enforced on every cluster |
| Teams and scopes | Team access to namespaces across clusters |
| Multi-cluster Gateway | One load balancer across clusters in several regions |
| Connect gateway | One authenticated path to every registered cluster |
| Rollout sequencing | Upgrades reach one group of clusters before the next |
gcloud container fleet memberships register prod-eu \
--gke-cluster=europe-west1/prod-eu \
--enable-workload-identity --project=fleet-host-prodFleets assume sameness: a namespace or service with the same name in two clusters is treated as the same thing. Keep production and non-production in separate fleets.
Safe patching and upgrading
| Control | Purpose |
|---|---|
| Release channels (Rapid, Regular, Stable, Extended) | Google upgrades clusters automatically at a chosen pace |
| Maintenance windows | Upgrades only happen in agreed hours |
| Maintenance exclusions | Pause upgrades during a freeze, such as a sales peak |
| Surge upgrades | maxSurge and maxUnavailable control how many nodes change at once |
| Blue-green node pool upgrades | Create new nodes, move workloads, keep the old pool for quick rollback |
| PodDisruptionBudgets | Limit how many replicas an upgrade evicts at once (GKE respects them for up to an hour per node) |
Put non-production clusters on a faster channel than production, or use rollout sequencing, so problems appear in staging first. For VMs, use VM Manager patch jobs and managed instance group rolling updates to a new instance template.
A worked example
A company has 12 GKE clusters across three regions with manifests copied per cluster. The engineer registers them in two fleets, nonprod and prod, and turns on Config Sync from one repository with Kustomize overlays.
Policy Controller blocks privileged pods everywhere. Nonprod clusters use the Regular channel and prod uses Stable with maintenance windows and an exclusion over Black Friday, and every pull request gets a namespace that is deleted on merge.
How the exam asks about this
- "The same network policies and RBAC must be applied to 40 GKE clusters and stay in sync." Register them in a fleet and use Config Sync from a Git repository.
- "Cluster upgrades must never happen during the end-of-quarter freeze." Add a maintenance exclusion for that period.
- "Each pull request needs a working preview URL without affecting live traffic." Deploy a tagged Cloud Run revision with no traffic.
What to remember for the exam
- One project per environment; differences live in small overlays, not copies.
- Ephemeral environments are created and destroyed by the pipeline, and labelled.
- Fleets group clusters; Config Sync and Policy Controller keep them consistent.
- Release channels plus maintenance windows and exclusions control GKE upgrades.
- Upgrade non-production first; use surge or blue-green node pool upgrades in production.
Turning off automatic upgrades to avoid surprises leaves clusters on versions that later reach end of support and are upgraded anyway. Control timing with channels, windows and exclusions instead.
Common mistakes
Copying full manifests per environment. Fixes reach one copy and not the others. Use a base with Kustomize overlays or Helm values.
Putting production and test clusters in one fleet. Fleet sameness then treats test and production namespaces as the same. Use separate fleets.
Upgrading every cluster at once. A bad version hits everything together. Stage upgrades with channels or rollout sequencing.
Check yourself
What is a GKE fleet?
A logical group of clusters, in a fleet host project, that are managed together with shared features such as Config Sync and Policy Controller.
How do you stop GKE upgrades during a busy business period?
Configure a maintenance exclusion covering that period, alongside maintenance windows for normal times.
How can a pull request get a preview on Cloud Run without receiving user traffic?
Deploy a revision with a tag and the no traffic option, which gives it its own URL while live traffic stays on the current revision.
Topic 5 of 6 in Bootstrapping and maintaining a Google Cloud organization
Test yourself on Bootstrapping and maintaining a Google Cloud organization
The full Google Cloud Professional Cloud DevOps Engineer question bank has 751 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.