Resilience Assessment, Testing, and Chaos Engineering

What it is

A DR plan that has never been tested is a guess. Assessment checks whether the design can meet its RTO and RPO on paper, and testing proves it by breaking things on purpose and timing the recovery.

AWS Resilience Hub handles assessment and recommendations. AWS Systems Manager runbooks turn recovery steps into automation, and AWS Fault Injection Service (AWS FIS) injects real faults so you can measure what happens.

How it works

AWS Resilience Hub assessments

You define an application in Resilience Hub from its resources, for example from AWS CloudFormation stacks, and attach a resiliency policy. The policy sets RTO and RPO targets for four disruption types.

An assessment estimates the workload RTO and RPO for each application component and reports "Policy met" or "Policy breached". For example, a 24-hour Region target is breached if the database has no cross-Region recovery method.

The resiliency score shows how closely the application follows the recommendations for its policy, alarms, standard operating procedures (SOPs) and tests. Drift detection flags components that met the policy in a previous assessment but no longer do, and resources that were added or removed.

Recovery procedures with Systems Manager

Resilience Hub recommends SOPs, which are prescriptive recovery steps for an outage or alarm. Implement them as Systems Manager Automation runbooks so recovery is repeatable and fast, not a wiki page someone reads at 3 a.m.

Typical runbook steps are promoting a replica, restoring from a recovery point, scaling a standby fleet and switching traffic. Each SOP should be tested and timed, because the timing is what proves the RTO.

AWS Fault Injection Service

AWS FIS runs fault injection experiments on real AWS resources, based on chaos engineering principles. An experiment template is the blueprint.

FIS can run multi-account experiments that target resources in other accounts. Resilience Hub can also recommend FIS experiments that inject a failure, check that alarms detect it and check that the SOPs recover the application.

Chaos engineering with a steady-state hypothesis

Chaos engineering is a controlled experiment, not random breakage.

1. Define steady state with business metrics, such as successful orders per minute and p99 latency.
2. State a hypothesis: "If one AZ loses its instances, steady state holds."
3. Run the smallest experiment that tests it, with stop conditions as guardrails.
4. Compare the results to steady state, fix weaknesses and repeat.

AWS recommends planning experiments and running them in a pre-production environment before production. Start with a small blast radius, such as a few tagged instances, and widen it as confidence grows.

Failover tests and measuring recovery time

A failover test should time each stage: detection, decision, traffic shift and data promotion. The total is the measured RTO, which you compare to the target.

• Route 53 health checks show how fast unhealthy endpoints are detected and DNS answers change.
• FIS faults such as stopping instances in one AZ test zonal recovery.
• ARC zonal shift tests moving traffic away from an AZ.
• AWS Elastic Disaster Recovery drills launch recovery instances without affecting replication.

Record the results and feed gaps back into the design, the runbooks and the next Resilience Hub assessment.

How the exam asks about this

• "The company must check continuously whether each application can meet its RTO and RPO targets and be told when changes break that." -> AWS Resilience Hub with a resiliency policy, scheduled assessments and drift detection.
• "Inject AZ-level failures in production but stop automatically if order error rates rise." -> An AWS FIS experiment template with a CloudWatch alarm as the stop condition.
• "Recovery steps must run the same way every time and be triggered from an alarm." -> Systems Manager Automation runbooks implementing the SOPs.