Incident and Event Response · Topic 4 of 5

Troubleshooting Deployments and Infrastructure

Diagnose failures in CloudFormation, CodeBuild, CodePipeline, CodeDeploy, Auto Scaling groups and ECS tasks using events, logs and the right CLI commands.

6 min

In a nutshell

Find the earliest failed event; later failures may be consequences.
Missing capability acknowledgements cause InsufficientCapabilitiesException.
continue-update-rollback with skipped resources unsticks UPDATE_ROLLBACK_FAILED.

What it is

Troubleshooting questions give you a symptom and ask for the most likely cause or the fastest way to find it. The skill is knowing where each service records its failure reason and the usual root causes.

A reliable method is to find the first failure, not the last. Later errors are often side effects, such as dependent resources reporting "cancelled".

How it works

CloudFormation

List stack events and look for the earliest FAILED status, then read its reason.

aws cloudformation describe-stack-events --stack-name web-prod \
  --query "StackEvents[?contains(ResourceStatus,'FAILED')].[Timestamp,LogicalResourceId,ResourceStatusReason]" \
  --output table

Symptom	Likely cause
`InsufficientCapabilitiesException`	The template creates IAM resources or uses macros and you did not acknowledge the capability
Resource already exists	A fixed physical name conflicts; let CloudFormation generate names
Failed to receive resource signals	`cfn-signal` never ran: check `/var/log/cfn-init.log` and `/var/log/cloud-init-output.log`
Stack hangs on a custom resource	The function did not send a response to the pre signed URL
`DELETE_FAILED`	A non empty S3 bucket or a resource with dependants such as a network interface
`UPDATE_ROLLBACK_FAILED`	A resource cannot return to its old state; continue the rollback and skip it
Access denied	The caller or the stack service role lacks permission for a resource

For a stuck rollback or delete, use these commands.

aws cloudformation continue-update-rollback --stack-name web-prod \
  --resources-to-skip LogicalResourceId
aws cloudformation delete-stack --stack-name web-prod \
  --retain-resources LogicalResourceId

To keep failed resources for inspection on create, disable rollback.

CodeBuild and CodePipeline

CodeBuild streams logs to CloudWatch Logs, and the build record lists phases such as DOWNLOAD_SOURCE, INSTALL, PRE_BUILD, BUILD, POST_BUILD and UPLOAD_ARTIFACTS, with the status of each. The failing phase narrows the cause quickly.

Symptom	Likely cause
Source download fails	Service role lacks access to S3 or the repository, or the connection is not authorized
Buildspec error	File missing, wrong name or invalid YAML
Dependency download times out	Project in a VPC without a NAT gateway or endpoints
Cannot connect to Docker daemon	Privileged mode is off
Cannot push to ECR	Missing ECR permissions on the build role, or no login step

In CodePipeline, open the failed action for its details link. Pipelines that never start usually have a missing or disabled trigger, a branch filter that does not match, or a source connection that is still pending. Cross account failures usually trace back to the artifact bucket policy, the KMS key policy or the role the deploy action assumes.

CodeDeploy

Use the deployment view and the instance logs.

aws deploy get-deployment --deployment-id d-EXAMPLE1
aws deploy list-deployment-targets --deployment-id d-EXAMPLE1

On Linux instances the agent log is at /var/log/aws/codedeploy-agent/codedeploy-agent.log, and script output is in the deployment logs folder under /opt/codedeploy-agent/deployment-root/deployment-logs/.

Symptom	Likely cause
Instance never starts the deployment	Agent not running, no instance profile, or no network path to CodeDeploy
Cannot download the bundle	Instance profile lacks S3 read access
`SCRIPT_FAILED`	A hook script returned a non zero exit code
`HEALTH_CONSTRAINTS`	Too many instances unhealthy for the configuration, such as minimum healthy hosts
Next deployment fails after a good one	The old `ApplicationStop` script is broken

Auto Scaling groups

The scaling activity history states exactly why a launch failed.

aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg --max-items 5

Typical causes are an AMI or launch template that no longer exists, a missing security group, no capacity for the instance type in an AZ, the account instance quota being reached, subnets that have run out of IP addresses, or an encrypted volume whose KMS key does not allow the Auto Scaling service linked role.

Instances that launch and are terminated again in a loop usually fail the load balancer health check. Look at the health check path and port, the security group between load balancer and instance, a bad user data script, and a grace period that is too short.

ECS and Fargate

A stopped task has a reason.

aws ecs describe-tasks --cluster prod --tasks <task-id> \
  --query "tasks[].[stoppedReason,containers[].reason]"

Message	Likely cause
`CannotPullContainerError`	Image or tag missing, no ECR permission on the task execution role, or no route to ECR
`ResourceInitializationError`	Cannot fetch secrets or set up networking, often a missing endpoint or permission
Essential container exited	The application crashed: read the container logs
Failed ELB health checks	Wrong port or path, or the security group blocks the load balancer
Cannot place task	No instance with enough CPU or memory, or a constraint cannot be met

Remember there are two roles. The task execution role pulls images and writes logs; the task role gives the application its own permissions.

What to remember for the exam

Find the earliest failed event; later failures may be consequences.
Missing capability acknowledgements cause InsufficientCapabilitiesException.
continue-update-rollback with skipped resources unsticks UPDATE_ROLLBACK_FAILED.
cfn-init and cloud-init-output logs explain missing signals.
CodeBuild failures in a VPC often mean no NAT or endpoint; Docker failures mean privileged mode.
CodeDeploy needs the agent, an instance profile with S3 access and working scripts.
Scaling activity history shows why an Auto Scaling group could not launch.
Launch and terminate loops point to failing load balancer health checks.
ECS stopped reasons separate image pull, secret, health and memory problems.
The task execution role differs from the task role.

Exam tip

When the question says "what is the fastest way to find why", choose the service's own history, such as stack events, scaling activities or stopped task reason, over adding new monitoring.

Common mistakes

Reading the last error. A cancelled dependent resource is a symptom, not the cause.

Mixing up the two ECS roles. A pull failure is an execution role problem, while an application access denied is a task role problem.

Blaming the code first. Most pipeline failures are permissions, network paths or missing configuration.

Watch out

Disabling rollback keeps failed resources running and billable, so delete the stack once you finish debugging.

Check yourself

Where do you see why an Auto Scaling group failed to launch an instance?
In the scaling activity history, which gives the status message for each activity.
What does CannotPullContainerError usually point to?
A missing image, missing ECR permission on the task execution role, or no network route to ECR.
How do you recover a stack stuck in UPDATE_ROLLBACK_FAILED?
Continue the update rollback, optionally skipping the resources that cannot be restored.

Topic 4 of 5 in Incident and Event Response

Test yourself on Incident and Event Response

The full AWS DOP-C02 question bank has 844 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.