Ensuring solution and operations excellence · Topic 3 of 3
Quality Control and Reliability Testing
Judging quality controls by outcomes such as change failure rate, and proving reliability with load testing, chaos engineering and pen testing.
- 4 min
- Judge controls by outcomes: change failure rate, incidents, SLO attainment, time to restore.
- DORA metrics balance speed and stability.
- Load test at multiples of peak, in staging, with synthetic data, against SLOs.
What it is
Quality control measures (tests, reviews, scans, approvals) are only worth their cost if they reduce problems users see. Architects evaluate them with outcome metrics and then prove reliability with deliberate tests: load testing, chaos engineering and penetration testing.
How it works
Evaluating quality control measures
| Metric | Measures | Can be gamed by writing more tests? |
|---|---|---|
| Change failure rate (share of releases needing a rollback or hotfix) | Problems that reached production | No |
| Incidents and SLO attainment | Reliability users experienced | No |
| Time to restore service | Recovery capability | No |
| Deployment frequency and lead time | Delivery speed | Somewhat |
| Test coverage, defects found in review | Activity of the controls | Yes |
The four delivery measures popularized by DORA research (deployment frequency, lead time for changes, change failure rate and time to restore service) show whether teams deliver quickly and safely. Compare before and after a new control to see if it helped.
When leadership wants proof that new controls reduced customer-visible problems, report the change failure rate or incident rate before and after, not test counts.
Load testing
| Practice | Why |
|---|---|
| Test at multiples of the expected peak | Real launches exceed forecasts |
| Use a staging environment sized like production | Production users must not bear the risk |
| Generate load with distributed workers (on GKE or Compute Engine) | One machine cannot simulate realistic traffic |
| Use synthetic data | Real customer data may not leave its region or environment |
| Measure against SLOs | Pass or fail on latency and error targets |
Chaos engineering
Chaos engineering injects controlled failures to confirm the system behaves as designed:
| Experiment | Proves |
|---|---|
| Drain and cordon the nodes of one zone during load | Workloads survive a zone loss |
| Kill Pods or VMs at random | Autohealing and replicas work |
| Inject latency or errors into a dependency | Timeouts, retries and fallbacks work |
| Fail over a database | RTO and RPO are real |
Start small in staging, define the expected outcome and a stop condition, then expand carefully. Never surprise production users.
Penetration testing
You can penetration test your own Google Cloud resources without notifying Google, provided you follow Google Cloud's Acceptable Use Policy and Terms of Service and test only what you own. Focus on your applications, configurations and identities. Use Web Security Scanner in Security Command Center for common web vulnerabilities, and review findings from Security Health Analytics.
Building reliability into the pipeline
- Run critical scenario tests (for example safety-critical simulations) on every change, the full suite nightly.
- Add load tests before big launches and after major architecture changes.
- Schedule game days for chaos experiments and DR failovers.
- Feed every finding into the backlog with an owner.
A worked scenario
Before a dealer launch, leadership wants evidence that order confirmations stay under two seconds at ten times the expected peak and that orders keep completing when a zone is lost. Dealers use production around the clock, and EU rules forbid copying real data out of the EU. Generate synthetic orders at ten times the peak from load workers in the EU against the staging environment, and drain one zone's nodes in staging while orders are submitted.
How the exam asks about this
- "Did the new tests and reviews reduce customer-visible problems?" Change failure rate before and after.
- "Prove the system survives a zone loss." A chaos experiment draining a zone in staging under load.
- "Security wants to test our web app for vulnerabilities." Penetration testing under the Acceptable Use Policy, plus Web Security Scanner.
What to remember for the exam
- Judge controls by outcomes: change failure rate, incidents, SLO attainment, time to restore.
- DORA metrics balance speed and stability.
- Load test at multiples of peak, in staging, with synthetic data, against SLOs.
- Chaos engineering proves resilience with controlled failures and stop conditions.
- Penetration testing your own resources needs no notice to Google, within the Acceptable Use Policy.
Load or chaos tests against production that customers use around the clock can cause the outage they were meant to prevent.
Common mistakes
Measuring test coverage as proof of quality.
Testing only at the expected peak.
Taking zone resilience on trust instead of testing it.
Check yourself
Which metric shows whether quality controls reduced problems in production?
The change failure rate, the share of releases that need a rollback or hotfix.
How do you prove a GKE workload survives a zone loss?
Drain and cordon one zone's nodes in staging while it serves load.
Do you need Google's permission to penetration test your own Google Cloud resources?
No, but you must follow the Acceptable Use Policy and Terms of Service.
Topic 3 of 3 in Ensuring solution and operations excellence
Test yourself on Ensuring solution and operations excellence
The full Google Cloud Professional Cloud Architect question bank has 877 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.