Automated Recovery and Failover

What it is

Choosing a DR strategy is half the job. The other half is automation, so that recovery happens fast, the same way every time, and without relying on someone remembering a long runbook at 3 in the morning.

Automation exists at three levels: self healing inside a Region (replace a failed instance), failover between AZs (database standby), and failover between Regions (traffic and data move to a second Region).

How it works

Self healing inside a Region

• Auto Scaling: the group replaces instances that fail health checks and keeps the desired capacity. A group of one instance across two AZs is a simple self healing pattern for a legacy server.
• ECS and EKS: the service scheduler restarts failed tasks, and the load balancer stops sending traffic to unhealthy ones.
• EC2 auto recovery: when the underlying host fails a system status check, EC2 can recover the instance onto new hardware. The recovered instance keeps its instance ID, private IP addresses, Elastic IP addresses and metadata. Do not rely on instance store data surviving, so keep important data on EBS.

You can attach the recover action to a CloudWatch alarm.

aws cloudwatch put-metric-alarm --alarm-name recover-web-1 \
  --namespace AWS/EC2 --metric-name StatusCheckFailed_System \
  --statistic Maximum --period 60 --evaluation-periods 2 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --alarm-actions arn:aws:automate:us-east-1:ec2:recover

The system status check points to AWS infrastructure trouble, while the instance status check points to the guest operating system. The recover action is for the first kind.

Database failover

RDS Multi AZ and Aurora fail over automatically inside the Region. For a Regional failure, Aurora Global Database supports a managed planned switchover with no data loss and an unplanned failover that promotes the secondary cluster, accepting possible loss of the last replicated changes. After promotion, the application needs the new writer endpoint, so use a Route 53 record, a configuration parameter or a proxy to hide the change.

DynamoDB global tables need no promotion because every replica accepts writes.

AWS Elastic Disaster Recovery

AWS Elastic Disaster Recovery (DRS) protects servers, including physical, virtual and cloud servers, by continuously replicating their disks at block level to a low cost staging area in the target AWS Region. At failover it launches recovery instances from the replicated data, with point in time options.

Choose it when the workload is a lifted server fleet, and it is cheaper than running a full warm standby.

Route 53 Application Recovery Controller

Amazon Route 53 Application Recovery Controller (ARC) coordinates failover for multi Region or multi AZ applications.

• Routing controls are on and off switches that change Route 53 health check results, so they steer traffic reliably through a highly available data plane.
• Safety rules prevent bad switch states, such as turning every routing control off at once.
• Zonal shift moves traffic away from an impaired AZ for supported resources such as load balancers, and zonal autoshift lets AWS start that shift for you.

Event driven recovery

Many recovery steps can be triggered by events.

Typical actions are restoring a database from the latest recovery point, promoting a replica, scaling up a pilot light fleet, updating a Parameter Store value, and switching a DNS record. A CloudFormation stack or StackSet can build the DR environment on demand.

Testing recovery

AWS Fault Injection Service (FIS) runs controlled experiments such as stopping instances, injecting network latency or throttling APIs, with stop conditions tied to alarms. Schedule game days to prove that failover meets RTO and RPO, and record the timings.