Deployment and Automation · Topic 2 of 7
CloudFormation Changes and Troubleshooting
Update CloudFormation stacks with change sets, detect drift, protect resources, fix failed rollbacks and deploy to many accounts with StackSets.
- 5 min
- Change sets preview an update, including replacements, before you execute it.
- Drift detection reports differences, and it does not fix them.
- A stack in ROLLBACK_COMPLETE after a failed create must be deleted.
What it is
Creating a stack is the easy part. The operations work is changing it safely, noticing when someone has changed resources by hand, and fixing stacks that get stuck.
This page covers the tools for that: change sets, drift detection, protection settings, rollback behaviour and StackSets for many accounts.
How it works
Change sets
A change set shows what an update will do before you do it. It lists each resource as added, modified or removed, and says whether a modification needs a replacement.
# AWS CLI
aws cloudformation create-change-set --stack-name web-stack \
--change-set-name add-cache --template-body file://web.yaml
aws cloudformation describe-change-set --stack-name web-stack \
--change-set-name add-cache
aws cloudformation execute-change-set --stack-name web-stack \
--change-set-name add-cacheUse change sets in production to avoid surprises, especially replacements of databases.
Drift detection
Drift means the real resource no longer matches the template, usually because of a manual change. Drift detection compares the two and reports each resource as in sync, modified or deleted.
aws cloudformation detect-stack-drift --stack-name web-stack
aws cloudformation describe-stack-resource-drifts --stack-name web-stackDrift detection reports and does not repair. To fix drift, change the resource back to match the template by hand, or update the template to match reality.
Protection settings
| Setting | What it does |
|---|---|
| Stack policy | JSON rules that block updates to chosen resources unless explicitly allowed |
| Termination protection | Stops the whole stack from being deleted by mistake |
| DeletionPolicy | Per resource: Delete, Retain or Snapshot when the stack is deleted |
| UpdateReplacePolicy | The same choices, applied when an update replaces the resource |
| Rollback triggers | CloudWatch alarms that cause a rollback if they fire during an update |
Use DeletionPolicy: Retain or Snapshot on databases and buckets that hold important data. Snapshot is available for resources that support snapshots, such as EBS volumes and RDS instances.
Stack states and failures
| State | Meaning and action |
|---|---|
| CREATE_FAILED then ROLLBACK_COMPLETE | Creation failed and was undone. The stack cannot be updated, so delete it and create again |
| UPDATE_ROLLBACK_COMPLETE | Update failed and the stack returned to the previous working state |
| UPDATE_ROLLBACK_FAILED | The rollback itself failed. Fix the cause, then continue the update rollback, skipping resources if needed |
| DELETE_FAILED | A resource could not be deleted, for example a non empty S3 bucket. Fix it and delete again, or retain it |
The Events tab of the stack shows the first failure reason, and later errors are often side effects. Common causes are missing permissions, a quota limit, an invalid parameter value and a name that already exists.
Turning off rollback on create keeps the failed resources, so you can inspect them. Do this for troubleshooting only.
Importing and moving resources
Resource import brings an existing resource under stack management without recreating it. This is how you adopt hand built resources into IaC.
StackSets
A StackSet deploys one template to many accounts and Regions in one operation. With AWS Organizations you can use service managed permissions, which also lets new accounts in a chosen organizational unit receive the stack automatically. With self managed permissions, you create the administrator and execution IAM roles yourself.
Typical uses are security baselines, AWS Config rules and logging setup across the organization.
Other provisioning tools
AWS Service Catalog publishes approved CloudFormation templates as products for governed self service (covered on the last page of this chapter). The AWS CDK (Cloud Development Kit) lets you write infrastructure in a programming language, and it produces CloudFormation templates.
"Preview the effect of an update" means a change set. "Find out whether someone changed a resource manually" means drift detection.
What to remember for the exam
- Change sets preview an update, including replacements, before you execute it.
- Drift detection reports differences, and it does not fix them.
- A stack in ROLLBACK_COMPLETE after a failed create must be deleted.
- UPDATE_ROLLBACK_FAILED is handled with continue update rollback.
- DeletionPolicy Retain or Snapshot protects data when a stack is deleted.
- A stack policy blocks unwanted updates to critical resources.
- Termination protection prevents accidental deletion of a stack.
- StackSets deploy a template to multiple accounts and Regions.
- Service managed StackSets use Organizations and can cover new accounts automatically.
Common mistakes
Fixing drift by making another console edit. The next template update can fight the change. Put the intended change into the template.
Reading the last event as the cause. The first failure in the event list is usually the real cause.
Expecting DeletionPolicy to protect against updates. Replacement during an update needs UpdateReplacePolicy.
Trying to update a ROLLBACK_COMPLETE stack. It is only fit for deletion.
Using a stack policy as access control. It guards against updates by anyone, and does not replace IAM permissions.
A stack delete removes resources that have the Delete policy, including databases created in the stack. Add Snapshot or Retain before the data matters.
Check yourself
Answer these three questions.
How can you see whether an update will replace a database before it runs?
Create a change set and read the replacement flag for that resource.
A new stack ends in ROLLBACK_COMPLETE. What now?
Read the first failure event, fix the cause, delete the stack and create it again.
Which feature deploys one template to many accounts and Regions?
AWS CloudFormation StackSets.
Topic 2 of 7 in Deployment and Automation
Test yourself on Deployment and Automation
The full AWS SOA-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.