Operational Excellence and Automation · Topic 3 of 7
Deployment Strategies and Failure Recovery
Pick blue/green, canary or rolling deployments with rollback, design health check replacement and add circuit breakers, retries and bulkheads.
- 6 min
- Blue/green and canary give the fastest rollback; rolling is cheapest.
- CodeDeploy rolls back automatically on deployment failure or on CloudWatch alarms.
- The ECS deployment circuit breaker can roll back a failing rolling update.
What it is
Every release and every failure is a moment when a workload can break. Deployment strategies control how new code reaches users, so a bad version touches few of them and rolls back on its own.
Failure recovery replaces broken instances, tasks or invocations automatically. Safety constraints such as circuit breakers, rate limits, bulkheads and backoff make sure that recovery automation cannot overload the system it is trying to fix.
How it works
Deployment strategies
| Strategy | How it works | Rollback speed | Extra cost |
|---|---|---|---|
| All at once | Replace everything in one step | Slow, redeploy old version | None |
| Rolling | Replace in batches | Slow, roll forward or back in batches | Low |
| Rolling with extra batch or immutable | New capacity first, then remove old | Medium | Temporary extra capacity |
| Blue/green | Full new environment, switch traffic | Fast, switch back | Double capacity during the switch |
| Canary | Small percentage first, then the rest | Fast | Low |
| Linear | Equal increments at fixed intervals | Fast | Low |
CodeDeploy supports canary, linear and all-at-once traffic shifting for Lambda aliases and Amazon ECS, and in-place or blue/green for EC2. It can roll back automatically when a deployment fails or a CloudWatch alarm fires.
Amazon ECS rolling updates can use the deployment circuit breaker, which marks a deployment failed when tasks keep failing to start and can roll back to the last completed deployment. ECS also has a built-in blue/green deployment type with a bake time and Lambda lifecycle hooks that can test the green revision before and after traffic shifts.
EC2 Auto Scaling instance refresh replaces instances in batches with a minimum healthy percentage, and can roll back automatically when it fails or an alarm triggers.
Other options: Lambda alias weighted routing, API Gateway canary release settings, and Route 53 weighted records for whole environments in different accounts or Regions.
"Fastest rollback" points to blue/green or canary. "Minimal extra cost" points to rolling. Blue/green with a database schema change needs backward compatible schemas, because both versions use the data at once.
Health checks and replacement
Automatic recovery depends on a health check that reflects the real application.
| Compute | Health signal | Replacement |
|---|---|---|
| EC2 Auto Scaling | EC2 status checks by default; add ELB health checks | Terminates and launches a new instance |
| Amazon ECS service | Container health check and load balancer target health | Scheduler stops the task and starts a new one |
| Amazon EKS | Liveness and readiness probes | Kubernetes restarts or reschedules pods |
| Lambda async | Function error | Retries, then sends to an on failure destination or DLQ |
| Single EC2 instance | Status check alarm | EC2 recover action keeps the instance ID and IP addresses |
Turn on the ELB health check type in the Auto Scaling group. Without it, an instance whose application has hung but whose OS is fine stays in service. Set a health check grace period so new instances are not killed while they boot.
For Lambda asynchronous invocations, configure the maximum retry attempts (0 to 2) and the maximum event age. Send events that still fail to an on failure destination such as SQS, SNS, Lambda or EventBridge. For stream sources, use bisect batch on error and an on failure destination so one poison record does not block the shard.
Safety constraints on remediation
Automation that retries too hard, or fixes everything at once, can turn a small fault into an outage.
| Pattern | Problem it prevents | AWS feature |
|---|---|---|
| Retry with exponential backoff and jitter | Retry storms against a struggling dependency | SDK standard and adaptive retry modes, Step Functions Retry with BackoffRate and jitter |
| Circuit breaker | Calling a dependency that keeps failing | ECS deployment circuit breaker, application logic that fails fast |
| Rate limit | One client or automation overwhelming a service | API Gateway throttling and usage plans, Systems Manager rate controls |
| Bulkhead | One failing part consuming all shared capacity | Lambda reserved concurrency, separate queues, cells or accounts |
| Dead letter queue | Poison messages retried forever | SQS redrive policy with maxReceiveCount |
Systems Manager Automation and Run Command take a concurrency value and an error threshold. The run stops when failures pass the threshold, so a broken runbook does not reach the whole fleet. Use FIS experiments to prove these safeguards work before an incident does.
How the exam asks about this
- "Shift 10 percent of Lambda traffic to the new version, then all of it if no errors after 10 minutes, with automatic rollback" -> CodeDeploy canary configuration on a Lambda alias with CloudWatch alarms.
- "Instances fail health checks on the load balancer but Auto Scaling never replaces them" -> enable the ELB health check type on the Auto Scaling group.
- "A remediation runbook must patch 2,000 instances but stop if more than a few fail" -> Systems Manager Automation with concurrency and error threshold rate controls.
What to remember for the exam
- Blue/green and canary give the fastest rollback; rolling is cheapest.
- CodeDeploy rolls back automatically on deployment failure or on CloudWatch alarms.
- The ECS deployment circuit breaker can roll back a failing rolling update.
- Auto Scaling needs the ELB health check type to replace application-level failures.
- Lambda async retries up to two times, then uses on failure destinations or a DLQ.
- Use exponential backoff with jitter, rate controls and bulkheads to keep recovery safe.
Retries without backoff, or immediate retries from thousands of clients, are a classic cause of a full outage after a short dependency failure. Pick the answer that adds jitter and a retry limit.
Common mistakes
Choosing blue/green when cost is the constraint. It needs a full second environment during the switch. A rolling or canary deployment is cheaper when budget is stated.
Relying on EC2 status checks only. They catch hardware and OS failures, not an application returning errors. Load balancer health checks are what detect those.
Letting automation act without limits. A remediation that runs everywhere at once spreads a mistake to the whole fleet. Concurrency limits, error thresholds and a canary account contain it.
Check yourself
Which deployment strategy gives instant rollback by switching traffic back?
Blue/green, because the old environment keeps running until the new one is proven.
How does an ECS service recover from a rolling deployment whose tasks keep failing?
The deployment circuit breaker marks the deployment failed and, with rollback enabled, returns the service to the last completed deployment.
What protects a struggling dependency from a storm of retries?
Exponential backoff with jitter and a capped number of attempts, optionally behind a circuit breaker.
Topic 3 of 7 in Operational Excellence and Automation
Test yourself on Operational Excellence and Automation
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.